**6. Simulation analyses**

The *OptSelectionAHP* provides meta‐heuristic approach for optimal selection problem, whose effectiveness and efficiency should be analyzed and evaluated. We have chosen the closeness to an optimal solution as the criterion which is used to estimate the efficiency of the approach.

Furthermore, the *OptSelectionAHP* approach provides an optimization of the approach from our previous work [20], which has been shown to be efficient with almost 90% of optimality applied in software engineering environment for the problem of optimal configuration of business process families. The optimization proposed with *OptSelectionAHP* approach is made in accordance with characteristics of introduced selection criteria structure (as described in Section 4). This is the reason why we decided to make simulation comparison of *OptSelectio‐ nAHP* approach with meta‐heuristic approach previously developed in reference [20].

Furthermore, domain of business process families is characterized with optimality and uncertainty in configuration process, as well with sets of integrity constraints, thus allowing to analyze how variability in selection space influence on optimality of proposed approach.

With this in mind, we defined the following hypothesis that we tested in our experiments.

H1. There is no significant difference between the distances to the optimal solution, compared to un‐optimized meta‐heuristic approach.

H2. In case of different distributions of optional elements, there is no significant difference between the distances to the optimal solution.

For testing the hypotheses and making an estimation of the average distance to an optimal solution, we performed several experiments which are explained in the next section.

#### **6.1. Experimental setup**

In order to perform the analyses, we separately performed two different experiments, described later in this section. Each experiment includes generation of business process families with parametrically changeable values of descriptive parameters [i.e., number of activities and their interconnections, available services, quality of services (as criteria for making configurations) and set of preferences]. In that sense, two generators are used similarly as in previous experiments [20]. Also, we use the brute‐force algorithm and previous GA adoption in order to obtain the optimal solution and solution with un‐optimized approach and compare them with the solution of *OptSelectionAHP*.

In our previous work [28], we suggested the number of seven qualifier tags in the two‐layered structure, as optimal number manageable by humans. We also set 100 options as maximum number of available options. All values are generated randomly. The four groups of selection criteria are considered (in domain of business processes configuration that are Quality of Services (QoS) attributes [29] with random number of characteristics to each group [22, 30].

No systematic parameter optimization process has so far been attempted, but we use the following parameters in our experiments: population size *P* =1, maximum generation *G* =200, crossover probability = 1 (always applied), mutation rate = 0.1, dynamic penalty factor, *w*(*gen*)=*C∙gen*, C = 0.5 [31].

#### *6.1.1. Experiment 1: Comparison of optimal and heuristic algorithms*

(step //7) Replace operator costs *O*(*P*); the validity of each element of the population is checked with the *optionsTransform* algorithm, which takes *T(optionsTransform*). Over each element of

significantly reduced complexity compared to our previous work [20] where the complexity

The *OptSelectionAHP* provides meta‐heuristic approach for optimal selection problem, whose effectiveness and efficiency should be analyzed and evaluated. We have chosen the closeness to an optimal solution as the criterion which is used to estimate the efficiency of the approach.

Furthermore, the *OptSelectionAHP* approach provides an optimization of the approach from our previous work [20], which has been shown to be efficient with almost 90% of optimality applied in software engineering environment for the problem of optimal configuration of business process families. The optimization proposed with *OptSelectionAHP* approach is made in accordance with characteristics of introduced selection criteria structure (as described in Section 4). This is the reason why we decided to make simulation comparison of *OptSelectio‐ nAHP* approach with meta‐heuristic approach previously developed in reference [20].

Furthermore, domain of business process families is characterized with optimality and uncertainty in configuration process, as well with sets of integrity constraints, thus allowing to analyze how variability in selection space influence on optimality of proposed approach.

With this in mind, we defined the following hypothesis that we tested in our experiments.

H1. There is no significant difference between the distances to the optimal solution, compared

H2. In case of different distributions of optional elements, there is no significant difference

For testing the hypotheses and making an estimation of the average distance to an optimal

In order to perform the analyses, we separately performed two different experiments, described later in this section. Each experiment includes generation of business process families with parametrically changeable values of descriptive parameters [i.e., number of activities and their interconnections, available services, quality of services (as criteria for making configurations) and set of preferences]. In that sense, two generators are used similarly

solution, we performed several experiments which are explained in the next section.

<sup>2</sup> <sup>+</sup> *<sup>n</sup><sup>∙</sup> <sup>s</sup>* <sup>2</sup>

<sup>2</sup> <sup>+</sup> *<sup>n</sup>* <sup>+</sup> *<sup>P</sup>* <sup>+</sup> *cnk∙log* <sup>2</sup>

<sup>2</sup> <sup>+</sup> *<sup>n</sup><sup>∙</sup> <sup>s</sup>* <sup>2</sup> 2 )

*k*)). This is

population, the selection criteria measurement is calculated, which takes *O*(*<sup>r</sup>* <sup>+</sup> *<sup>k</sup>* <sup>2</sup>

208 Applications and Theory of Analytic Hierarchy Process - Decision Making for Strategic Decisions

Thus, the iteration steps of the GAs take *OGA* <sup>=</sup>*O*(*G*(*<sup>r</sup>* <sup>+</sup> *<sup>k</sup>* <sup>2</sup>

was exponential to the size of selection criteria model.

operations as given earlier.

**6. Simulation analyses**

to un‐optimized meta‐heuristic approach.

**6.1. Experimental setup**

between the distances to the optimal solution.

Our main goal in this experiment is to estimate differences between qualities of solutions as a measure of how close our proposed algorithm is to an optimal solution. Also, we analyze whether optimization issues significantly influence the optimality of the approach (defined as H1).

The approximation ratio (heuristic utility vs. the optimal value) is used as an appropriate metric of closeness to the optimal solution. Random generations of business process families and appropriate selection criteria models (QoS attributes) are performed 1000 times; the optimization goals are solved simultaneously with both heuristic approaches while a brute‐ force algorithm is used for obtaining the optimal solution.

Given the type of the collected data in the simulations, we analyzed them with standard descriptive statistics including mean (M) and standard deviation (SD) values. The hypothesis is tested by ANOVA for comparing means for multiple independent populations to see if a significant difference exists in the distances to the optimal solution in cases of using optimized approach compared to previous un‐optimized approach (H1).

**Experimental results.** Calculated mean value of relative distance between configurations obtained by *OptSelectionAHP* and optimal configurations is equal to 10.41% (SD = 0.964%). Thus, the optimality of our approach is around 89.5%.

Furthermore, the mean value of the results obtained by un‐optimized approach for optimal configuration, the same collection of business process families is equal to 10.20% (SD = 0.928%). Even if the mean values of both approaches are close, graphical representation (shown in **Figure 5**) shows incoherence between optimality of two approaches. Only in 23.00% (line 2), both approaches have the same precision, while in 13.4% (line 1) and 17.2% (line 3), the solutions of both approaches are optimal.

**Figure 5.** Scatter diagram of relations between distances to optimal solution of meta‐heuristic approaches.

As the collected data were not normally distributed, a one way ANOVA test was used over the log‐transformed data to compare the means of obtained relative distances to the optimal solution. The results show a non‐significant difference between approaches F(1,782) = 11.814, *p* = 0.229. Thus, hypothesis H1 is accepted and we can conclude that the optimization issues do not have a significant impact on the optimality of the approach. This finding is important, since the complexity of the whole approach is reduced to polynomial without loss of accuracy.

#### *6.1.2. Experiment 2: Analyses of the performance characteristics in the cases of different characteristics of input parameters*

For testing the hypothesis H2, we performed several simulations with different distributions of optional elements in the business process families. Distributions of optional elements are determined with the percentage ratio and we considered the following values: 25%, 50%, 75% and 100% (these are referred to as groups 1–4, respectively). Each simulation is performed 1000 times and the collected data are used for testing the hypotheses with ANOVA for comparing means for multiple independent populations to see if a significant difference exists in the distances to the optimal solution.

**Experimental results.** The results show a non‐significant difference between approaches related to different distributions of optional elements in business process families: F(3,584) = 28.489, *p* = 0.171; mean values for each group are presented in **Table 1**.


**Table 1.** Mean values of relative distance between solutions of *OptSelectionAHP* compared to optimal solution for different distributions of optimal elements in business process model.

**Experiment conclusion**. The observed results show that variability elements do not represent a source of statistical impact on optimality of obtained results. Hence, hypothesis H2 is accepted. In *OptSelectionAHP* approach, dynamic penalties in the fitness function are defined on the basis of qualitative measurements and identified characteristics of different kinds of preferences, therefore proving the importance of dynamic penalties for convergence of the whole approach.

#### *6.1.3. Discussion*

**Figure 5.** Scatter diagram of relations between distances to optimal solution of meta‐heuristic approaches.

210 Applications and Theory of Analytic Hierarchy Process - Decision Making for Strategic Decisions

As the collected data were not normally distributed, a one way ANOVA test was used over the log‐transformed data to compare the means of obtained relative distances to the optimal solution. The results show a non‐significant difference between approaches F(1,782) = 11.814, *p* = 0.229. Thus, hypothesis H1 is accepted and we can conclude that the optimization issues do not have a significant impact on the optimality of the approach. This finding is important, since the complexity of the whole approach is reduced to polynomial without loss of accuracy.

*6.1.2. Experiment 2: Analyses of the performance characteristics in the cases of different characteristics*

For testing the hypothesis H2, we performed several simulations with different distributions of optional elements in the business process families. Distributions of optional elements are determined with the percentage ratio and we considered the following values: 25%, 50%, 75% and 100% (these are referred to as groups 1–4, respectively). Each simulation is performed 1000 times and the collected data are used for testing the hypotheses with ANOVA for comparing means for multiple independent populations to see if a significant difference exists in the

**Experimental results.** The results show a non‐significant difference between approaches related to different distributions of optional elements in business process families: F(3,584) =

28.489, *p* = 0.171; mean values for each group are presented in **Table 1**.

*of input parameters*

distances to the optimal solution.

Experimental results show that proposed *OptSelectionAHP* approach has proven optimality for optimal selection goals. Furthermore, its convergence guided by dynamic penalties has shown good characteristic even in cases of different distributions of optional elements in the experiment.
