**7. Evaluation**

### **7.1. Overview evaluation**

To investigate the performance, accuracy and limitations of the proposed system and to compare both search strategies (DFS, BFS) as well as to discuss the optimal parameters for the masking area and VFOV, the results of 63 experiments from two setups are presented.

### **7.2. First setup**

The first setup contained 42 experiments. The search area consisted of a 3 m × 2 m square with two randomly placed balls at the positions (50, 50) and (240, 140) according to **Figure 16**.

**Figure 16.** First setup.

In this setup the experiment was repeated for both search strategies, BFS and DFS, for four different masking areas with MA = 0.1 m, MA = 0.15 m, MA = 0.2 m and MA = 0.3 m and with three different VFOV: 0.3 m × 0.45 m, 0.40 m × 0.60 m and 0.60 m × 0.90 m. Then the computed position of the target was compared with the manually measured one, supposed to be the true position. For every single parameter setting, the average error d*x* in the *X*‐ and d*y* in the *Y*‐ direction was computed, first over both targets and then over the entire run together. Also the number of double detections D (fail positive) and misses M (fail negative) was counted (**Table 2**). In the second run, the experiments for MA = 0.3 m were skipped because MA = 0.2 m showed no problem in this setup.


**Table 2.** Detection errors first setup: M missing and D double detections.

From these data no clear difference in accuracy between DFS and BFS or between the different parameter settings could be identified, but it could be concluded that the average error in one axis is less than 15 cm. This setup of randomly placed balls is predominantly affected by coincidence. It might be that one setting leading to one flight path fits well to the placement of the balls.

By taking a look at the detection failures (**Table 2**), clear conclusions can be made. The real FOV is about 65 cm × 45 cm and it can clearly be seen that a VFOV of 40 cm × 60 cm or higher leads to misses. The bigger the VFOV is, the more misses occur, as expected. A proper VFOV of 30 cm × 45 cm leads to no misses for both search strategies. The data show that a lower MA can lead to double detections. This is the case because a target might be seen several times. As the position error in one direction is about 15 cm, MA should be at least in the same range. Conclusively, it can be seen that the DFS performed better and also that there is still a domi‐ nating systematical error.

### **7.3. Second setup**

device. The kernel of RODOS provides support for thread execution, time management, synchronization and transparent access to external devices such as sensors by the HAL

To investigate the performance, accuracy and limitations of the proposed system and to compare both search strategies (DFS, BFS) as well as to discuss the optimal parameters for the masking area and VFOV, the results of 63 experiments from two setups are presented.

The first setup contained 42 experiments. The search area consisted of a 3 m × 2 m square with two randomly placed balls at the positions (50, 50) and (240, 140) according to **Figure 16**.

In this setup the experiment was repeated for both search strategies, BFS and DFS, for four different masking areas with MA = 0.1 m, MA = 0.15 m, MA = 0.2 m and MA = 0.3 m and with three different VFOV: 0.3 m × 0.45 m, 0.40 m × 0.60 m and 0.60 m × 0.90 m. Then the computed position of the target was compared with the manually measured one, supposed to be the true position. For every single parameter setting, the average error d*x* in the *X*‐ and d*y* in the *Y*‐ direction was computed, first over both targets and then over the entire run together. Also the number of double detections D (fail positive) and misses M (fail negative) was counted (**Table 2**). In the second run, the experiments for MA = 0.3 m were skipped because MA = 0.2

(hardware abstraction layer) interface.

**7. Evaluation**

**7.2. First setup**

**Figure 16.** First setup.

m showed no problem in this setup.

**7.1. Overview evaluation**

18 Recent Advances in Robotic Systems

Based on the outcome of the first setup, in the second setup more balls were placed to reduce the effect of coincidence. In addition, the search area was changed to a 2 m × 3 m square (**Figures 17** and **18**), which aimed to equalize the results between the two search strategies and to improve the results of the BFS. This time positions were selected, which should cause troubles for all settings (**Table 3**).

**Figure 17.** Second setup.

**Figure 18.** Picture of second setup from above.


**Table 3.** Detection errors second setup: M missing and D double detections.

**Figure 19** depicts the results shown in the QT Control‐Software for a run with the settings MA = 20 cm and 30 × 45 cm for VFOV. It demonstrates that for these settings all targets were detected properly.

The second setup more clearly showed the effect of each parameter or setting and underlined the already expected results. More targets reduced the effect of coincidence, and therefore one run was seen to be enough.

The average position error for the DFS was 16 cm and for the BFS it was about 20 cm. According to these data the DFS can already be concluded as more accurate, but a clearer distinction between both search strategies can be made by taking the detection failures into account. For the DFS there are 10 detection errors in nine experiments compared to 20 detections errors of the BFS in the same setup. Considering these bad results, a value of 25 cm × 35 cm for VFOV was tested with the BFS, but this led to even worse results. There is no setting for the BFS without detection error, but there are four settings with no detection error for the DFS.

**Figure 19.** GUI picture of search result (20–30–45): red: found targets; green: waypoints; yellow: position.

### **7.4. Summary evaluation**

**Figure 18.** Picture of second setup from above.

20 Recent Advances in Robotic Systems

**Detection failures DFS BFS MA VFOV M D M D 15 25–35 Skipped 2 4**

**20 25–35 Skipped 3 2**

**30 25–35 Skipped 2 0**

**Figure 19** depicts the results shown in the QT Control‐Software for a run with the settings MA = 20 cm and 30 × 45 cm for VFOV. It demonstrates that for these settings all targets were detected

The second setup more clearly showed the effect of each parameter or setting and underlined the already expected results. More targets reduced the effect of coincidence, and therefore one

The average position error for the DFS was 16 cm and for the BFS it was about 20 cm. According to these data the DFS can already be concluded as more accurate, but a clearer distinction between both search strategies can be made by taking the detection failures into account. For the DFS there are 10 detection errors in nine experiments compared to 20 detections errors of the BFS in the same setup. Considering these bad results, a value of 25 cm × 35 cm for VFOV

**Table 3.** Detection errors second setup: M missing and D double detections.

properly.

run was seen to be enough.

**30–45 0 2 2 0 40–60 0 0 1 0 60–90 2 0 4 0**

**30–45 0 0 1 0 40–60 0 0 2 0 60–90 2 0 4 0**

**30–45 0 0 1 0 40–60 1 0 1 0 60–90 3 0 4 0** To sum up, it can be said that all settings, the search strategy, the masking area and the VFOV have a significant effect on the performance of the search. Although still other, partly random parameters and circumstances have an important influence on the result, optimal values of these parameters are required. This is underlined by **Figures 20**–**22**, which show that the DFS with MA = 20 cm and a VFOV of 30 cm × 45 cm or 40 cm x 60 cm detected eight balls exactly and nothing else mistakenly. This means there exist settings, which solved this challenging setup. It shall be mentioned that an MA of 30 cm led to a miss in one of the two cases because

**Figure 20.** DFS detection failures (general distribution).

**Figure 21.** BFS detection failures (distribution after MA).

then one of the closest balls, which are 20 cm away, was not accepted. The BFS showed no good results here and only a VFOV of 30 cm × 45 cm and of 40 cm × 60 cm led to acceptable results. Altogether these results also made the BFS look worse than it was. Some balls were positioned in such way that the BFS failed them by a few centimetres.

**Figure 22.** BFS detection failures (distribution after VFOV).
