**8.1 Classification accuracy**

From the perspective of the classification accuracy, there are four parameters could be discussed which are overall accuracy, kappa coefficient, user's and producer's accuracies (analysis per class). These parameters can be calculated from error matrix tables. A classification error matrix was computed for quantitative accuracy assessment. Table 4, 5, 6 and 7 demonstrated the error matrices table deriving from MD, ML, NN and FBC classifier. The dominant land cover types in the selected area were urban, mountain and land areas which correspond to 95% of the entire image. The remaining 5% of the image is consisted by vegetation, ritual area and shadow.

For MD algorithm which is the simplest classifier among others, the result of overall accuracy was 64.2% with 0.479 value of kappa coefficient was obtained. The user accuracy is varied between 50.7% for urban class and 100.0% for vegetation and ritual area classes. Mountain, land and shadow classes recorded 79.6%, 62.2% and 73.1% respectively. For producer accuracy, the accuracy for each class using MD approach was as follow: 67.1% for urban, 57.9% for mountain, 70.9% for land, 45.5% for vegetation (lowest), 66.7% for ritual area and 95.0% for Shadow (highest). A total of 500 random sample points were tested in order to verify the classification result with 321 points was correctly classified. Meanwhile, urban class recorded almost half of the tested pixels that correctly classified with most of the misclassified pixel go to mountain class. A total of 129 out of 162 observations had been correctly classified for mountain class and 33 points were wrongly classified with 26 points were misclassified as urban class. The high number of wrongly pixels go to urban class is due to the fact that the mountainous area in the arid environment is not cover by tree but it is filled by stones and rocks which is has a similar spectral characteristic of urban area. Nevertheless, vegetation and ritual area classes gave the perfect result by correctly classified all tested points. Both classes are easily to classify due to the significantly different on their spectral characteristics among other classes. In the other hand, 56 out of 90 observations for

accuracy, which indicates the proportion of ground base reference samples correctly assigned. It details errors of omission, i.e., when a pixel is omitted from its correct category. The other error is given by the user's accuracy, which indicates the proportion of data from the estimation map representing that category on the ground. It is a measure of errors of commission, i.e., when a pixel is committed to an incorrect category (Avelar et al., 2009).

The six classes-urban, mountain, land, vegetation, ritual area and shadow were classified using four different classifiers, and classification accuracy assessments were conducted (Table 4-7). Performances of each of the classifiers that have been tested will be analyzed based on three factors. In order to make a comparison, the classifiers performance are analyse in term of their classification accuracy, training samples and performance in heterogeneous area. The area of each class estimated through various techniques was compared and evaluated with the corresponding actual area as obtained from the reference data. For lack of additional satellite data, concurrent with the periods of the field surveys, a reference dataset was generated based on the ordered SPOT-5 satellite data and expert

From the perspective of the classification accuracy, there are four parameters could be discussed which are overall accuracy, kappa coefficient, user's and producer's accuracies (analysis per class). These parameters can be calculated from error matrix tables. A classification error matrix was computed for quantitative accuracy assessment. Table 4, 5, 6 and 7 demonstrated the error matrices table deriving from MD, ML, NN and FBC classifier. The dominant land cover types in the selected area were urban, mountain and land areas which correspond to 95% of the entire image. The remaining 5% of the image is consisted by

For MD algorithm which is the simplest classifier among others, the result of overall accuracy was 64.2% with 0.479 value of kappa coefficient was obtained. The user accuracy is varied between 50.7% for urban class and 100.0% for vegetation and ritual area classes. Mountain, land and shadow classes recorded 79.6%, 62.2% and 73.1% respectively. For producer accuracy, the accuracy for each class using MD approach was as follow: 67.1% for urban, 57.9% for mountain, 70.9% for land, 45.5% for vegetation (lowest), 66.7% for ritual area and 95.0% for Shadow (highest). A total of 500 random sample points were tested in order to verify the classification result with 321 points was correctly classified. Meanwhile, urban class recorded almost half of the tested pixels that correctly classified with most of the misclassified pixel go to mountain class. A total of 129 out of 162 observations had been correctly classified for mountain class and 33 points were wrongly classified with 26 points were misclassified as urban class. The high number of wrongly pixels go to urban class is due to the fact that the mountainous area in the arid environment is not cover by tree but it is filled by stones and rocks which is has a similar spectral characteristic of urban area. Nevertheless, vegetation and ritual area classes gave the perfect result by correctly classified all tested points. Both classes are easily to classify due to the significantly different on their spectral characteristics among other classes. In the other hand, 56 out of 90 observations for

**8. Performance evaluation** 

**8.1 Classification accuracy** 

vegetation, ritual area and shadow.

knowledge.

land class were correctly classified whereas 19 out of 26 observations for shadow class were correctly classified. Most of the incorrect pixels were classified as mountain class. This is not surprising because most of the shadow appear within mountainous area.

Meanwhile, ML algorithm was the second traditional method that has been tested in this project. It gave better result than MD classifier. The overall accuracy was 77.6% while the kappa coefficient had a value of 0.659. For each classes (user accuracy), urban recorded 69.8%, 83.4% for mountain, 82.9% for land, 92.6% for vegetation, 66.7% for ritual area (lowest) and 100.0% for shadow (highest). Although overall classification result was better than MD, but two classes (vegetation and ritual area) showing lower percentage than MD. In the meantime, producer accuracy is varied between 61.5% (shadow) and 100.0% (ritual area). Urban, mountain, land and vegetation classes had a value of 85.4%, 69.6%, 73.3% and 96.2% respectively. Further evaluation of the error matrix shows that 388 out of 500 points used from the same random samples were correctly classified. The classifier had some difficulty separating cleared land from land under construction (urban) and mountain from urban area, as exhibited by error matrix table that showed 68 points were wrongly classified to both classes (53 points for mountain, 15 points for land). This is understandable because their spectral characteristics are very similar. However, the result of urban class revealed that significant improvement (nearly 20%) was achieved compared to the MD classifier. In


Table 4. Error matrix derived from Minimum Distance-to-Mean classifier


Table 5. Error matrix derived from Maximum Likelihood classifier

Analysis of Land Cover Classification

77.1%, 93.8% and 92.3% respectively.

in Arid Environment: A Comparison Performance of Four Classifiers 131

classifiers for this factor. The overall accuracy was 84.2% and had a value of 0.757 for kappa coefficient. The NN method seems to do a much better job in classifying all classes than the other methods, which seems to be the primary reason for its high overall accuracy. The success of this classifier is due to the fact that four classes (land, vegetation, ritual area and shadow) have been tested correctly perfect (100% is obtained in analysis per class). In fact, the mountain class achieved a high user's accuracy (95.3 percent) using NN method with 162 out of 170 observations were correctly classified. Nevertheless, urban class recoded 72.5% in analysis per class but it still acceptable and highest among other classes. From the view point of statistical analysis, most of the pixels in urban area were confuse with mountain (48 points) and land (23 points). The reason why each of classifier always resulted urban class to the lower percentage compared to other classes will be explained in the next sub-section. Meanwhile, producer's accuracy varied between 59.7% for land class and 100% for ritual area class. Urban, mountain, vegetation and shadow classes recorded 98.9%,

The network architecture for the NN had three layers, with twelve units in the hidden layer, four units in the input layer (one for each spectral band), and six units in the output layer (one for each class). The other parameters used in the NN algorithm are shown in Table 8. These network structures were determined through trial and error meaning the number of hidden units used in this application was determined through experimental simulations. Fig. 6 shows the variation of RMSE values at convergence as a function of the number of hidden nodes. The experiments were performed with a maximum number of iterations of 1000 and the final RMSE was between 0.009 and 0.109 with the number of hidden nodes ranging from 3 to 15 nodes with increment of 3. The minimum RMSE with the smallest number of nodes was attained adopting architecture with 12 hidden nodes. This was the

architecture finally adopted for the learning and classification process.

Table 8. Parameters used in the neural network algorithm

Fig. 6. Graph of RMSE versus Number of Nodes

NN architecture 4-12-6 Momentum rate 0.9 Learning rate 0.1

Iteration 1000 epochs


Table 6. Error matrix derived from Back-propagation Neural Network classifier


\* Note: Urb = Urban, Mou = Mountain, Lan = Land, Veg = Vegetation, Rit = Ritual area, Sha = Shadow and Unk = Unknown.

Table 7. Error matrix derived from Frequency-based Contextual classifier

addition, 126 out of 151 observations were correctly classified for mountain class whereas 63 out of 76 observations were correctly classified for land class. Most of the misclassification for both classes goes to urban class. For vegetation class, although it has lower percentage over MD classifier, but the result still to be considered as a good result by obtaining over 90% with only 2 out of 27 observations were wrongly classified. Ritual area class was another category that showed their percentage lower than MD classifier. Although the class was easily to classify but they recorded only 66.7% when validation process was performed.

The lower in accuracy for the ritual area class is explained by the fact that only 3 out of 500 points were tested in that particular class meaning insufficient validation points occurred in this class. The result is expected to be higher if more validation point is added during the validation process as this class was a homogenous category. In the other hand, shadow class gave a perfect result by correctly classified all tested pixels.

However, NN approach which was one of the advanced methods tested in this project demonstrated superior result in term of overall accuracy. The NN outperformed the other

Urb **187** 48 23 0 0 0 258 72.4 Mou 2 **162** 4 1 0 1 170 95.3 Lan 0 0 **40** 0 0 0 40 100.0 Veg 0 0 0 **15** 0 0 15 100.0 Rit 0 0 0 0 **5** 0 5 100.0 Sha 0 0 0 0 0 **12** 12 100.0

Total 189 210 67 16 5 13 **500** 

Table 6. Error matrix derived from Back-propagation Neural Network classifier

Unk 2 3 1 0 0 0 **6**  Total 183 214 75 7 6 15 **500** 

PA (%) 89.1 77.1 77.3 28.6 100.0 93.3

Table 7. Error matrix derived from Frequency-based Contextual classifier

gave a perfect result by correctly classified all tested pixels.

PA (%) 98.9 77.1 59.7 93.8 100.0 92.3

Overall accuracy = 84.2% Kappa coefficient = 0.757

Overall accuracy = 81.6% Kappa coefficient = 0.722

and Unk = Unknown.

Urb Mou Lan Veg Rit Sha Total UA (%)

Urb Mou Lan Veg Rit Sha Total UA (%)

Urb **163** 45 15 5 0 1 229 71.2 Mou 13 **165** 0 0 0 0 178 92.7 Lan 2 0 **58** 0 0 0 60 96.7 Veg 0 0 0 **2** 0 0 2 100.0 Rit 3 0 1 0 **6** 0 10 60.0 Sha 0 1 0 0 0 **14** 15 93.3

\* Note: Urb = Urban, Mou = Mountain, Lan = Land, Veg = Vegetation, Rit = Ritual area, Sha = Shadow

addition, 126 out of 151 observations were correctly classified for mountain class whereas 63 out of 76 observations were correctly classified for land class. Most of the misclassification for both classes goes to urban class. For vegetation class, although it has lower percentage over MD classifier, but the result still to be considered as a good result by obtaining over 90% with only 2 out of 27 observations were wrongly classified. Ritual area class was another category that showed their percentage lower than MD classifier. Although the class was easily to classify but they recorded only 66.7% when validation process was performed. The lower in accuracy for the ritual area class is explained by the fact that only 3 out of 500 points were tested in that particular class meaning insufficient validation points occurred in this class. The result is expected to be higher if more validation point is added during the validation process as this class was a homogenous category. In the other hand, shadow class

However, NN approach which was one of the advanced methods tested in this project demonstrated superior result in term of overall accuracy. The NN outperformed the other classifiers for this factor. The overall accuracy was 84.2% and had a value of 0.757 for kappa coefficient. The NN method seems to do a much better job in classifying all classes than the other methods, which seems to be the primary reason for its high overall accuracy. The success of this classifier is due to the fact that four classes (land, vegetation, ritual area and shadow) have been tested correctly perfect (100% is obtained in analysis per class). In fact, the mountain class achieved a high user's accuracy (95.3 percent) using NN method with 162 out of 170 observations were correctly classified. Nevertheless, urban class recoded 72.5% in analysis per class but it still acceptable and highest among other classes. From the view point of statistical analysis, most of the pixels in urban area were confuse with mountain (48 points) and land (23 points). The reason why each of classifier always resulted urban class to the lower percentage compared to other classes will be explained in the next sub-section. Meanwhile, producer's accuracy varied between 59.7% for land class and 100% for ritual area class. Urban, mountain, vegetation and shadow classes recorded 98.9%, 77.1%, 93.8% and 92.3% respectively.

The network architecture for the NN had three layers, with twelve units in the hidden layer, four units in the input layer (one for each spectral band), and six units in the output layer (one for each class). The other parameters used in the NN algorithm are shown in Table 8. These network structures were determined through trial and error meaning the number of hidden units used in this application was determined through experimental simulations. Fig. 6 shows the variation of RMSE values at convergence as a function of the number of hidden nodes. The experiments were performed with a maximum number of iterations of 1000 and the final RMSE was between 0.009 and 0.109 with the number of hidden nodes ranging from 3 to 15 nodes with increment of 3. The minimum RMSE with the smallest number of nodes was attained adopting architecture with 12 hidden nodes. This was the architecture finally adopted for the learning and classification process.


Table 8. Parameters used in the neural network algorithm

Fig. 6. Graph of RMSE versus Number of Nodes

Analysis of Land Cover Classification

compared to the same classifiers.

**8.2 Performances in heterogeneous area** 

in Arid Environment: A Comparison Performance of Four Classifiers 133

In general, the NN approach generally provided the highest accuracies for all classes. Considering the overall accuracy, NN provided the best classification results with 84.2% and MD provided the poorest results with overall accuracy of 64.2%. Fig. 8 provides a comparison of kappa and overall accuracy results among the different classifiers. It indicates that NN and FBC have a significantly better accuracy than do MD and ML classifiers. MD produced lower classification accuracy because it only used the mean vector and ignored the covariance between the classes. ML produced a relatively higher accuracy than did MD because it takes the covariance into account in its algorithm. However, ML assumes a normal distribution for the histograms of the classes, which is not always true. Both MD and ML only consider per-pixel information, ignoring texture or contextual information. Comparing the two approaches (traditional and advanced methods), the proposed NN classifier proved to be more effective, with a 6.6% and 20.0% increase in accuracy compared to ML and MD classifier whereas FBC could increased their accuracy up to 4.0% and 17.4%

Fig. 8. Comparison of overall accuracy and kappa coefficient using different classifier

The heterogeneity environment in an image is a major problem in classification where a pixel contains more than one land cover class. In our case, urban class is considered as a heterogeneous area instead of homogenous area for remaining of the five other classes. In this section, we will explain the reason why the percentage of urban class in this work is always lower compared to other classes. This is due to the urban factor itself. The spectral characteristics of urban surfaces are known to be complex. This is due to the fact that much information could be extracted from the urban class. Urban areas are characterized by a large variety of built-up environments and natural vegetation covers which not only determine the surface features of a city, such as land use patterns, but also influence ecological, climatic and energetic conditions of land surface processes (Chen et al, 2009). For instance, sites under construction possess a more varied high reflectance resulting from building construction foundations and construction materials. Cleared land exhibits high uniform spectral reflectance which is characteristic of bare soil, while some vegetated area also located in urban environment. These numbers of information in a single class would

For FBC classifier, it is required to determine window sizes before begin the classification process. This window size was very important because it will impact on how much the spatial information will be included during decision making process. But the exactly size is not easy to determine. Hence, it was determined by experiment. From the experimental simulations, 9x9 window size was determined as an optimum window. But we decided to choose window sizes of 7x7 instead of 9x9 as the images used for other three classifiers were filter out using 7x7 averaging filter to reduce the noisy effect from the original image. Moreover, there are no significant differences on overall accuracy between these two windows as well as for the remaining of window sizes beyond 9x9 as shown in Fig 7. The FBC techniques used in this project achieved higher overall accuracy and kappa coefficient (81.6 percent and 0.722) rather than traditional method. Even though it cannot overtake the performance of NN but their result is still good and acceptable. Further evaluation of the error matrix shows that the additional of contextual information increases map accuracy. The high quality of the spatial information had a large impact on the success of this method. However, there are some extremely difficult types of confusion to map. Desert landscaping often consist of gravel, and certain types of gravel can be spectrally indistinguishable from urban. To make the confusion even more complex, some part of mountainous area were located within urban area. This situation would lead the misclassification to be occurred since their spectral characteristic is similar. In the meantime, the class specific producer accuracy varied between 28.6% for vegetation class and 100.0% for ritual area class. User accuracy reached the highest value of 100.0% for vegetation class. Lowest values were obtained for the class ritual area with 60.0%. The integration of contextual information showed its benefits in the sharp improvement in accuracy for the mountain and land classes compared to traditional method. Other behavior of FBC method that it can be seen from the classification result that pixels at the edge of different land cover type are mostly misclassified. At the center of each land cover type, most classes are correctly classified. By evaluating error matrix table, it revealed that urban and mountain classes were confused each other. Shadow, land and vegetation classes were easily classified with all observation points were classified correctly for vegetation class. Nevertheless, the unexpected result was achieved by ritual area class where it gave lower result (60 percent) although this class was considered as homogenous area with has uniformly in their spectral characteristic. The sharp decrease in accuracy for that class is explained by the fact that it was not suitable to use the current window sizes due to the homogenous behavior of the class. For this situation, smaller window size is more suitable and could be expected to increase the class accuracy.

Fig. 7. Graph showing experimental result of FBC using different window sizes

For FBC classifier, it is required to determine window sizes before begin the classification process. This window size was very important because it will impact on how much the spatial information will be included during decision making process. But the exactly size is not easy to determine. Hence, it was determined by experiment. From the experimental simulations, 9x9 window size was determined as an optimum window. But we decided to choose window sizes of 7x7 instead of 9x9 as the images used for other three classifiers were filter out using 7x7 averaging filter to reduce the noisy effect from the original image. Moreover, there are no significant differences on overall accuracy between these two windows as well as for the remaining of window sizes beyond 9x9 as shown in Fig 7. The FBC techniques used in this project achieved higher overall accuracy and kappa coefficient (81.6 percent and 0.722) rather than traditional method. Even though it cannot overtake the performance of NN but their result is still good and acceptable. Further evaluation of the error matrix shows that the additional of contextual information increases map accuracy. The high quality of the spatial information had a large impact on the success of this method. However, there are some extremely difficult types of confusion to map. Desert landscaping often consist of gravel, and certain types of gravel can be spectrally indistinguishable from urban. To make the confusion even more complex, some part of mountainous area were located within urban area. This situation would lead the misclassification to be occurred since their spectral characteristic is similar. In the meantime, the class specific producer accuracy varied between 28.6% for vegetation class and 100.0% for ritual area class. User accuracy reached the highest value of 100.0% for vegetation class. Lowest values were obtained for the class ritual area with 60.0%. The integration of contextual information showed its benefits in the sharp improvement in accuracy for the mountain and land classes compared to traditional method. Other behavior of FBC method that it can be seen from the classification result that pixels at the edge of different land cover type are mostly misclassified. At the center of each land cover type, most classes are correctly classified. By evaluating error matrix table, it revealed that urban and mountain classes were confused each other. Shadow, land and vegetation classes were easily classified with all observation points were classified correctly for vegetation class. Nevertheless, the unexpected result was achieved by ritual area class where it gave lower result (60 percent) although this class was considered as homogenous area with has uniformly in their spectral characteristic. The sharp decrease in accuracy for that class is explained by the fact that it was not suitable to use the current window sizes due to the homogenous behavior of the class. For this situation, smaller window size is more suitable and

could be expected to increase the class accuracy.

Fig. 7. Graph showing experimental result of FBC using different window sizes

In general, the NN approach generally provided the highest accuracies for all classes. Considering the overall accuracy, NN provided the best classification results with 84.2% and MD provided the poorest results with overall accuracy of 64.2%. Fig. 8 provides a comparison of kappa and overall accuracy results among the different classifiers. It indicates that NN and FBC have a significantly better accuracy than do MD and ML classifiers. MD produced lower classification accuracy because it only used the mean vector and ignored the covariance between the classes. ML produced a relatively higher accuracy than did MD because it takes the covariance into account in its algorithm. However, ML assumes a normal distribution for the histograms of the classes, which is not always true. Both MD and ML only consider per-pixel information, ignoring texture or contextual information. Comparing the two approaches (traditional and advanced methods), the proposed NN classifier proved to be more effective, with a 6.6% and 20.0% increase in accuracy compared to ML and MD classifier whereas FBC could increased their accuracy up to 4.0% and 17.4% compared to the same classifiers.

Fig. 8. Comparison of overall accuracy and kappa coefficient using different classifier
