**6.2 Experiment 1: Evaluation of** *m***-Medoids based frameworks for classification and anomaly detection**

The purpose of this experiment is to evaluate the performance of proposed MMC-GFS and LMC-ES based frameworks for classification of unseen data samples to one of the known patterns. The effectiveness of the proposed frameworks to perform anomaly detection is also demonstrated here. The experiment has been conducted on simulated SIM2 dataset. Training

Fig. 4. Classification of test data, based on SIM2 classes, using (a) LMC-ES framework (b)

**6.3 Experiment 2: Comparison of proposed classifiers with competitive techniques**

The purpose of this experiment is to compare the performance of classifiers as proposed in LMC-ES and MMC-GFS frameworks. For comparison of our results with competitive techniques, we establish a base case by implementing three different systems for comparison including Mahalanobis and GMM classifier. Real life ASL dataset is used for the experiment.

4. Classification obtained by applying the MMC-GFS approach on LAB dataset is shown in Fig. 7. The matching of classification obtained for each trajectory with its ground truth shows that no trajectory is misclassified. Trajectories identified as anomalous are shown in Fig. 8. It is clear from Fig. 8 that anomalous trajectories are significantly different from the normal motion patterns as shown in Fig. 7. The classification experiment is also conducted on the HIGHWAY dataset and the results obtained are shown in Fig. 9. Trajectories filtered as anomalous are shown in Fig. 10. These experimental results give evidence to the claim that MMC-GFS based classification and anomaly detection system is an effective and robust approach that works

Behavior Recognition Using any Feature Space Representation of Motion Trajectories 113

MMC-GFS framework.

well with real life motion datasets.

data from simulated datasets is shown in Fig. 3. Test data for SIM2 dataset is obtained by generating 500 samples from a uniform distribution such that (*x*, *y*) ∈ (*U*(1, 12), *U*(1, 12)).

We have used 50 medoids to model a class using its member samples. The classification and anomaly detection results for SIM2 dataset, using LMC-ES and MMC-GFS frameworks are presented in Fig. 4(a) and Fig. 4(b) respectively. Training samples are represented using '+' marker whereas classified normal samples are represented by small circles. Data points belonging to same class are represented with same colour and marker. Samples from test data which have been identified as anomalous are represented with a black 'x' marker. It is apparent from Fig. 4 that multimodal *m*-Medoids based classification system as proposed in MMC-GFS framework performs better classification and anomaly detection while catering for multimodal distribution within the modeled pattern. On the other hand, LMC-ES based framework performs unimodal modeling of patterns and therefore the classification system does not adjust well to the variation of density within a pattern.

After demonstrating the efficacy of proposed classification and anomaly detection approach on synthetic data, the experiment is then repeated on real life LAB and HIGHWAY datasets. LAB and HIGHWAY datasets are classified motion datasets and contain anomalous trajectories within the datasets themselves. Classified training data for these datasets is obtained by randomly selecting half of the trajectories from each of the normal patterns in the dataset. The remaining half of the trajectories from the normal patterns along with anomalous trajectories are extracted and used as test data. Training samples from the LAB and HIGHWAY datasets are shown in Fig. 5 and Fig. 6 respectively. For ease of visualization, samples from each class are plotted separately on the background scene. The starting point of each trajectory is marked in green.

Trajectories from LAB and HIGHWAY datasets are modelled using DFT-MOD based coefficient feature vectors. (Khalid, 2010b). Patterns are modeled using 20 medoids per pattern. Once the multimodal *m*-Medoids based model for all the classes have been learnt, classification of samples from the test data is done using the classifier as proposed in section

data from simulated datasets is shown in Fig. 3. Test data for SIM2 dataset is obtained by generating 500 samples from a uniform distribution such that (*x*, *y*) ∈ (*U*(1, 12), *U*(1, 12)). We have used 50 medoids to model a class using its member samples. The classification and anomaly detection results for SIM2 dataset, using LMC-ES and MMC-GFS frameworks are presented in Fig. 4(a) and Fig. 4(b) respectively. Training samples are represented using '+' marker whereas classified normal samples are represented by small circles. Data points belonging to same class are represented with same colour and marker. Samples from test data which have been identified as anomalous are represented with a black 'x' marker. It is apparent from Fig. 4 that multimodal *m*-Medoids based classification system as proposed in MMC-GFS framework performs better classification and anomaly detection while catering for multimodal distribution within the modeled pattern. On the other hand, LMC-ES based framework performs unimodal modeling of patterns and therefore the classification system

After demonstrating the efficacy of proposed classification and anomaly detection approach on synthetic data, the experiment is then repeated on real life LAB and HIGHWAY datasets. LAB and HIGHWAY datasets are classified motion datasets and contain anomalous trajectories within the datasets themselves. Classified training data for these datasets is obtained by randomly selecting half of the trajectories from each of the normal patterns in the dataset. The remaining half of the trajectories from the normal patterns along with anomalous trajectories are extracted and used as test data. Training samples from the LAB and HIGHWAY datasets are shown in Fig. 5 and Fig. 6 respectively. For ease of visualization, samples from each class are plotted separately on the background scene. The starting point of each trajectory

Trajectories from LAB and HIGHWAY datasets are modelled using DFT-MOD based coefficient feature vectors. (Khalid, 2010b). Patterns are modeled using 20 medoids per pattern. Once the multimodal *m*-Medoids based model for all the classes have been learnt, classification of samples from the test data is done using the classifier as proposed in section

Fig. 3. Training data from SIM2 dataset.

is marked in green.

does not adjust well to the variation of density within a pattern.

Fig. 4. Classification of test data, based on SIM2 classes, using (a) LMC-ES framework (b) MMC-GFS framework.

4. Classification obtained by applying the MMC-GFS approach on LAB dataset is shown in Fig. 7. The matching of classification obtained for each trajectory with its ground truth shows that no trajectory is misclassified. Trajectories identified as anomalous are shown in Fig. 8. It is clear from Fig. 8 that anomalous trajectories are significantly different from the normal motion patterns as shown in Fig. 7. The classification experiment is also conducted on the HIGHWAY dataset and the results obtained are shown in Fig. 9. Trajectories filtered as anomalous are shown in Fig. 10. These experimental results give evidence to the claim that MMC-GFS based classification and anomaly detection system is an effective and robust approach that works well with real life motion datasets.

#### **6.3 Experiment 2: Comparison of proposed classifiers with competitive techniques**

The purpose of this experiment is to compare the performance of classifiers as proposed in LMC-ES and MMC-GFS frameworks. For comparison of our results with competitive techniques, we establish a base case by implementing three different systems for comparison including Mahalanobis and GMM classifier. Real life ASL dataset is used for the experiment.

Fig. 6. Labelled training samples from HIGHWAY dataset. Trajectories belonging to different

Behavior Recognition Using any Feature Space Representation of Motion Trajectories 115

**MMC-GFS** 0.99 0.94 0.91 0.86 0.83 **LMC-ES** 0.98 0.92 0.88 0.83 0.78 **Mahalanobis** 0.95 0.88 0.82 0.75 0.71 **GMM** 0.97 0.92 0.83 0.74 0.69

of classes as compared with competitive techniques; thus making them more scalable for larger number of classes. The superior performance of MMC-GFS, as compared to competitive techniques, can be explained by the fact that the proposed multimodal *m*-Medoids based frameworks do not impose any restriction on the probability distribution function of modeled patterns. The proposed frameworks can effectively model arbitrary shaped patterns and can effectively handle variation in sample distribution within a pattern as demonstrated in Fig. 1 and Fig. 4. On the other hand, the competitive approaches impose assumptions on the PDF of patterns (normally gaussian). These approaches do not have the capacity to handle multimodal distribution within a pattern. As a result, the model generated by these approaches will not give an accurate representation of complex patterns and hence result in poor classification performance as compared to the proposed multimodal *m*-Medoids based

Table 2. Classification accuracies for different number of classes from ASL dataset

**ASL (#classes : #samples) 2 : 70 4 : 140 8 : 280 16 : 560 24:840**

classes are plotted separately on background scene

approaches.

Fig. 5. Labelled training samples from LAB dataset. Trajectories belonging to different classes are plotted separately on background scene

Signs from different number of word classes are selected. Classified training data is obtained by randomly selecting half of the trajectories from each word class leaving the other half to be used as test data. Trajectories from ASL dataset are represented using DFT-MOD based coefficient feature vectors (Khalid, 2010b). Patterns are modeled using 20 medoids per pattern. We have computed single multimodal Gaussian for modeling of patterns for Mahalanobis classifier. Modeling of patterns and classification of unseen samples using GMM is based on the approach as described in (Bashir et al., 2005a). Each class is modeled using a separate GMM. The number of modes to be used for GMM-based modeling is automatically estimated using a string of pruning, merging and mode-splitting processes as specified in (Bashir et al., 2005a). Once the models for all the classes have been learnt, the test data is passed to different classifiers and the class labels obtained are compared with the ground truth. The experiment is repeated with different numbers and combinations of word classes. Each classification experiment is averaged over 50 runs to reduce any bias resulting from favorable word selection.

The accuracy of different classifiers for wide range of word classes from ASL dataset is presented in Table 2. Based on these results, we can see that the multimodal *m*-Medoids based classifier as proposed in MMC-GFS framework yield a superior classification accuracy as compared to other classifiers closely followed by unimodal LMC-ES framework. GMM yields good results for lower number of classes but its performance deteriorates for higher number of word classes. It can also be observed from Table 2 that the relative accuracy of proposed *m*-Medoids based MMC-GFS and LMC-ES classifiers increases with an increase in the number

Fig. 5. Labelled training samples from LAB dataset. Trajectories belonging to different classes

Signs from different number of word classes are selected. Classified training data is obtained by randomly selecting half of the trajectories from each word class leaving the other half to be used as test data. Trajectories from ASL dataset are represented using DFT-MOD based coefficient feature vectors (Khalid, 2010b). Patterns are modeled using 20 medoids per pattern. We have computed single multimodal Gaussian for modeling of patterns for Mahalanobis classifier. Modeling of patterns and classification of unseen samples using GMM is based on the approach as described in (Bashir et al., 2005a). Each class is modeled using a separate GMM. The number of modes to be used for GMM-based modeling is automatically estimated using a string of pruning, merging and mode-splitting processes as specified in (Bashir et al., 2005a). Once the models for all the classes have been learnt, the test data is passed to different classifiers and the class labels obtained are compared with the ground truth. The experiment is repeated with different numbers and combinations of word classes. Each classification experiment is averaged over 50 runs to reduce any bias resulting from

The accuracy of different classifiers for wide range of word classes from ASL dataset is presented in Table 2. Based on these results, we can see that the multimodal *m*-Medoids based classifier as proposed in MMC-GFS framework yield a superior classification accuracy as compared to other classifiers closely followed by unimodal LMC-ES framework. GMM yields good results for lower number of classes but its performance deteriorates for higher number of word classes. It can also be observed from Table 2 that the relative accuracy of proposed *m*-Medoids based MMC-GFS and LMC-ES classifiers increases with an increase in the number

are plotted separately on background scene

favorable word selection.

Fig. 6. Labelled training samples from HIGHWAY dataset. Trajectories belonging to different classes are plotted separately on background scene


Table 2. Classification accuracies for different number of classes from ASL dataset

of classes as compared with competitive techniques; thus making them more scalable for larger number of classes. The superior performance of MMC-GFS, as compared to competitive techniques, can be explained by the fact that the proposed multimodal *m*-Medoids based frameworks do not impose any restriction on the probability distribution function of modeled patterns. The proposed frameworks can effectively model arbitrary shaped patterns and can effectively handle variation in sample distribution within a pattern as demonstrated in Fig. 1 and Fig. 4. On the other hand, the competitive approaches impose assumptions on the PDF of patterns (normally gaussian). These approaches do not have the capacity to handle multimodal distribution within a pattern. As a result, the model generated by these approaches will not give an accurate representation of complex patterns and hence result in poor classification performance as compared to the proposed multimodal *m*-Medoids based approaches.

Fig. 9. Classification of test trajectories from HIGHWAY dataset

Behavior Recognition Using any Feature Space Representation of Motion Trajectories 117

Fig. 10. Trajectories identified as anomalous from HIGHWAY dataset using proposed

anomaly detection mechanism.

Fig. 7. Classification of test trajectories from LAB dataset

Fig. 8. Trajectories identified as anomalous from LAB dataset using proposed anomaly detection mechanism.

Fig. 7. Classification of test trajectories from LAB dataset

detection mechanism.

Fig. 8. Trajectories identified as anomalous from LAB dataset using proposed anomaly

Fig. 9. Classification of test trajectories from HIGHWAY dataset

Fig. 10. Trajectories identified as anomalous from HIGHWAY dataset using proposed anomaly detection mechanism.

and to the filtration of abnormal samples during the model generation phase. On the other hand, OCC-SVM generates good model of normal classes but classifies many of the abnormal samples as member of normal classes whereas GWR gives extra importance to filtering

Behavior Recognition Using any Feature Space Representation of Motion Trajectories 119

abnormal samples and in the process, identifies many normal samples as abnormal.

Fig. 11. Percentage anomaly detection accuracies for different number of classes from ASL

In this chapter, we have presented an extended *m*-Medoids based framework, referred to as MMC-GFS, for modeling of trajectory-based motion patterns. The strength of the proposed approach is its ability to model complex patterns without imposing any restriction on the distribution of samples within a given pattern. Once the multimodal *m*-Medoids model for all the classes have been learnt, the classification of new trajectories and anomaly detection is then performed using a proposed soft classification and anomaly detection algorithm which is adaptive to multimodal distributions of samples within a pattern. The strength of this technique is its ability to model complex patterns without imposing any restriction on the shape of patterns. MMC-GFS can be used for modeling, classification and anomaly detection

Experimental results are presented to show the effectiveness of proposed MMC-GFS classifier. Modeling of pattern and classification using proposed frameworks is unaffected by variation of sample distribution within a pattern as demonstrated in Fig. 4. Quantitative comparison of MMC-GFS based classifiers with competitive techniques demonstrates the superiority of our multimodal approach as it performs consistently better than commonly used Mahalanobis,

dataset

**7. Discussion and conclusions**

GMM and HMM-based classifiers.

in any feature space with a computable similarity function.

Similar experiment with ASL dataset (using similar experimental settings) has been conducted by (Bashir et al., 2007) using their proposed GMM and HMM-based classification system. They reported classification accuracies of 0.96, 0.92, 0.86 and 0.78 for 2, 4, 8 and 16 word classes respectively. A comparison of these classification accuracies with the results obtained using our approach reveals that classifiers from *m*-Medoids classifier family performs better than GMM and HMM-based recognition system (Bashir et al., 2007) despite the fact that our proposed classification approach is conceptually simpler and computationally less expensive.
