4. Fuzzy model of software reliability and security evaluation-based on test results

Testing of software complexes for compliance with requirements for reliability and security is one of the most time-consuming and difficult stages of implementation of automation system. This is primarily due to the extreme structural complexity of modern software and its heterogeneity. Incomplete information on the software structure, principles and functioning, heterogeneity of its composition, presence of imported elements, and insufficient specifications make it difficult to evaluate and predict the software reliability. In these cases, traditional approaches to acquisition and forecasting of reliable values are associated with significant costs; that is why models based on the fuzzy sets of theory that allow estimating the software reliability with practically acceptable accuracy are of immediate interest [26–28].

At the present time, the literature describes fuzzy models of software reliability evaluation. These models are peculiar for their focus on static and dynamic analysis of the software graph, which is practically difficult due to the extreme structural complexity of the modern software systems and environments. We suggest describing the software testing and debugging process by a non-monotonic software reliability growth function utilizing the fuzzy sets of theory in order to take into account the incompleteness of input data.

Prq ¼ P<sup>∞</sup> � ð Þ P<sup>∞</sup> � Pu

J ¼

formula for the number of remaining errors after u revision:

0 @

� � � � � �

Nu ¼

with practically acceptable accuracy are of immediate interest [26–28].

and j is the quantity of planned revisions.

160 Probabilistic Modeling in System Engineering

condition Puþ<sup>j</sup> ≥ Prg is met, the cycle stops.

software revision efficiency factor.

results

Y<sup>u</sup>þ<sup>j</sup> i¼uþ1

where Prq is the required level of the software reliability, u is the number of the last revision,

The quantity of revisions required to achieve the desired level of reliability can be calculated using the cyclic recalculation of the expression (Eq. (25)). To this end Pu is calculated using the formula (Eq. (25)); further, in the cycle the value Puþ<sup>j</sup> is defined by increasing j. When the

To simplify application of the predictive model, let us assume that Aj ¼ a, which corresponds to the transition from the model (Eq. (6)) to (Eq. (3)). Then, after we reduce the expression (Eq. (25)) and take its logarithm, we will obtain the following expression required to evaluate the number J of software revisions that are necessary to achieve the desired level of reliability:

> ln <sup>P</sup>∞�Prg P∞�Pu � �

1 A

1 A

� � � � � �

� � � � � �

ln 1ð Þ � a=P<sup>∞</sup>

where k k ℵ is the operation of obtaining of the nearest biggest integer ℵ and a is the averaged

Assuming that revisions do not introduce additional errors (i.e., P<sup>∞</sup> ¼ 1), we can obtain the

4. Fuzzy model of software reliability and security evaluation-based on test

Testing of software complexes for compliance with requirements for reliability and security is one of the most time-consuming and difficult stages of implementation of automation system. This is primarily due to the extreme structural complexity of modern software and its heterogeneity. Incomplete information on the software structure, principles and functioning, heterogeneity of its composition, presence of imported elements, and insufficient specifications make it difficult to evaluate and predict the software reliability. In these cases, traditional approaches to acquisition and forecasting of reliable values are associated with significant costs; that is why models based on the fuzzy sets of theory that allow estimating the software reliability

At the present time, the literature describes fuzzy models of software reliability evaluation. These models are peculiar for their focus on static and dynamic analysis of the software graph, which is practically difficult due to the extreme structural complexity of the modern software systems and environments. We suggest describing the software testing and debugging process

0 @

� � � � � � ln <sup>1</sup>�Prq 1�Pu � �

ln 1ð Þ � a

1 � P<sup>e</sup> <sup>i</sup>¼<sup>1</sup> aikij P<sup>∞</sup> � �

, (25)

, (26)

: (27)

It is possible to demonstrate that the non-monotonic software reliability growth function looks as follows:

$$P\_n = P\_{\text{ov}} - (P\_{\text{ov}} - P\_0) \left( 1 - \frac{a}{P\_{\text{ov}}} \right)^n \tag{28}$$

where Pn is the probability of successful software run after n revision, a is the revision efficiency factor, P<sup>0</sup> is the initial level of reliability, and P<sup>∞</sup> is the maximum level of reliability.

This model depends on three parameters that can be conveniently calculated with the help of the maximum likelihood method. To create the likelihood function, it is reasonable to use the data recorded during the software tests, namely, the order of revisions, results of the software runs (whether any vulnerabilities were detected or not), and number of runs between the revisions.

It is easy to show that the maximum likelihood function logarithm will look as follows:

$$\ln\left(L\_{n}\right) = \sum\_{j=1}^{n} \left(\widehat{m\_{j}}\ln\left(1 - P\_{\circ\circ} + (P\_{\circ\circ} - P\_{0})(1 - a/P\_{\circ\circ})^{i}\right)\right)$$

$$+ (n\_{\circ} - \widehat{m\_{\circ}}) \cdot \ln\left(P\_{\circ\circ} + (P\_{\circ\circ} - P\_{0})(1 - a/P\_{\circ\circ})^{i}\right).$$

where <sup>m</sup>c<sup>j</sup> is the number of failures in nj tests and <sup>n</sup> is the number of revisions.

The function ln ð Þ Ln is convex and is defined for a convex set; that is why in order to effectively find the maximum of the likelihood function we can use, for example, the modified steepest descent method with the variable increment parameter, which allows obtaining the desired parameters of the model (Eq. (28)). The greatest difficulty of modeling the automation system operational readiness is determined by the fact that the software reliability level has to be evaluated in conditions of considerable uncertainty, namely:


These uncertainties introduce a significant portion of subjectivity to the software reliability evaluation. The fuzzy set of theory allows taking them into account without substantial alteration of the model (Eq. (3)). This work is primarily aimed at solving this task.

#### 4.1. Development of a fuzzy software reliability and security model

Let us present the information on the debugging process in the form of the set X ¼ f g xi , where xi is the software revision (i ¼ 1, n). The number of relevant revisions is defined as <sup>m</sup> <sup>¼</sup> <sup>P</sup><sup>n</sup> <sup>i</sup>¼<sup>1</sup> χ<sup>i</sup> , where χ<sup>i</sup> ={0,1} is the characteristic function defining the presence of revision xi. Let us formalize the probable fuzziness of the software revision by transition from the characteristic function {0,1} to continuum [0,1]. Then, we have:


In this case, the fuzzy set of relevant revisions will look as follows:

$$\mathbf{M} = \{ \left( m, \mu\_M(m) \right) \}, \tag{29}$$

It is important to note that considering the monotonic dependence of the software reliability level from the number of revisions, it is possible to formalize the fuzzy set P (Eq. (34)) with the complex of hierarchically ordered crisp sets. According to the decomposition theorem, we have:

Then, by defining the value α based on the specific software operating conditions and accuracy

Below is the simplest example of calculation of the software reliability level. During the debugging stage, 48 tests were carried out, 5 groups of defects were detected, and required revisions were performed. After the expert opinions were processed, the information on

Having arranged the fuzzy set A by the membership function values, we obtained a fuzzy set

After we calculated reliability levels using the formulae (Eq. (3)), we obtained a fuzzy subset of

Thus, practical solutions suggested in the work take into account the uncertainties of software development and testing conditions. This allows obtaining rather accurate maximum and interval estimates of the software reliability and security. Analytical expressions allow simplifying the software reliability analysis as compared with the methods based on expert judgments. It is

It should be noted that there is no universal model of the software evaluation and test planning. Moreover, beside the described classes of models, studies suggest simulation models [29],

reasonable to apply the described results for planning of system and complex tests.

5. Evaluation models and test planning selection criteria

of expert estimation, we can obtain the interval (guaranteed) software reliability level:

αμ<sup>P</sup><sup>α</sup>

� �, (35)

Models for Testing Modifiable Systems http://dx.doi.org/10.5772/intechopen.75126 163

<sup>Ρ</sup> <sup>¼</sup> Pm <sup>j</sup> <sup>μ</sup>Mð Þ <sup>m</sup> <sup>≥</sup> <sup>α</sup> � �: (36)

A ¼ f g ð Þ 1; 0:0 ;ð Þ 2; 0:4 ;ð Þ 3; 0:2 ;ð Þ 4; 1:0 ;ð Þ 5; 0:9 : (37)

M ¼ f g ð Þ 0; 0:0 ;ð Þ 1; 0:2 ;ð Þ 2; 0:4 ;ð Þ 3; 0:6 ;ð Þ 4; 0:1 ;ð Þ 5; 0:0 : (38)

P ¼ f g ð Þ 0:31; 0:2 ;ð Þ 0:69; 0:4 ;ð Þ 0:97; 0:6 ;ð Þ 0:98; 0:1 : (39)

P ¼ ½ � 0:69; 0:97 : (40)

μ<sup>P</sup> ¼ ⋃ α∈ ½ � 0;1

where <sup>μ</sup><sup>P</sup><sup>α</sup> <sup>¼</sup> <sup>0</sup>,если <sup>μ</sup>ð Þ<sup>x</sup> <sup>≥</sup> <sup>α</sup>;

(

of relevant revisions:

the software reliability levels:

1,если μð Þx < α:

4.2. Example of possible application of fuzzy sets

debugging was obtained in the form of a fuzzy set of revisions:

According to the accepted assurance level α=0.4, we have.

where μMð Þ m is the membership function defining the level of confidence in the fact that the number of relevant revisions is equal to m.

In general, the membership function can be found using the following expression:

$$\mu\_M = \max\min\left\{ \overline{\mu\_{i\_1}}, \dots \overline{\mu\_{i\_m}}, \mu\_{j\_1}, \dots, \mu\_{j\_{(n-m)}} \right\}. \tag{30}$$

For the purpose of practical calculation, it is convenient to expand the revision membership function in ascending and descending order:

$$\begin{cases} \mu\_0 \ge \mu\_1 \ge \dots \ge \mu\_m \ge \mu\_{m+1} \ge \dots \ge \mu\_n; \\\\ \overline{\mu\_0} \le \overline{\mu\_1} \le \dots \le \overline{\mu\_m} \le \overline{\mu\_{m+1}} \le \dots \le \overline{\mu\_n}. \end{cases} \tag{31}$$

This provides the main calculated ratio: <sup>μ</sup>Mð Þ¼ <sup>m</sup> min <sup>μ</sup><sup>m</sup>þ<sup>1</sup> � , <sup>μ</sup>mÞ: The number of relevant revisions corresponding to the maximum level of confidence (i.e., to the maximum membership function) is equal to:

$$m = \sum\_{i=0}^{n} m\_{i\prime} \tag{32}$$

where mj <sup>¼</sup> <sup>0</sup>, если <sup>μ</sup><sup>i</sup> <sup>&</sup>lt; <sup>μ</sup><sup>i</sup> ; 1, если μ<sup>i</sup> ≥ μ<sup>i</sup> : (

The maximum membership function can be calculated in the following way:

$$\mu\_{\max} = \min\_{1 < i < m} \max\_{1 < i < m} \left( \mu\_i, \overline{\mu\_i} \right). \tag{33}$$

By applying the generalization principle, we can move from the fuzzy set of relevant revisions (Eq. (29)) to the desired fuzzy set of the software reliability levels:

$$P = \{ \left( P\_m, \mu\_p(P\_m) \right) \},\tag{34}$$

where <sup>μ</sup>Pð Þ¼ Pm min <sup>μ</sup><sup>i</sup>þ<sup>1</sup> � , <sup>μ</sup><sup>i</sup> Þ, m ¼ 0, n; and Pm —reliability level defined according to the formula (Eq. (3)).

It is important to note that considering the monotonic dependence of the software reliability level from the number of revisions, it is possible to formalize the fuzzy set P (Eq. (34)) with the complex of hierarchically ordered crisp sets. According to the decomposition theorem, we have:

$$\mu\_{\mathcal{P}} = \bigcup\_{\alpha \in [0, 1]} \left( \alpha \mu\_{\mathcal{P}\_{\alpha}} \right) \tag{35}$$

where <sup>μ</sup><sup>P</sup><sup>α</sup> <sup>¼</sup> <sup>0</sup>,если <sup>μ</sup>ð Þ<sup>x</sup> <sup>≥</sup> <sup>α</sup>; 1,если μð Þx < α: (

<sup>m</sup> <sup>¼</sup> <sup>P</sup><sup>n</sup>

<sup>i</sup>¼<sup>1</sup> χ<sup>i</sup>

, where χ<sup>i</sup>

162 Probabilistic Modeling in System Engineering

1. Fuzzy set A ¼ хi; μAð Þ х<sup>i</sup>

teristic function {0,1} to continuum [0,1]. Then, we have:

In this case, the fuzzy set of relevant revisions will look as follows:

2. Set of relevant revisions R ¼ f g m , m ¼ 0, n.

number of relevant revisions is equal to m.

function in ascending and descending order:

ship function) is equal to:

where mj <sup>¼</sup> <sup>0</sup>, если <sup>μ</sup><sup>i</sup> <sup>&</sup>lt; <sup>μ</sup><sup>i</sup>

(

where <sup>μ</sup>Pð Þ¼ Pm min <sup>μ</sup><sup>i</sup>þ<sup>1</sup>

formula (Eq. (3)).

1, если μ<sup>i</sup> ≥ μ<sup>i</sup>

8 >><

>>:

This provides the main calculated ratio: <sup>μ</sup>Mð Þ¼ <sup>m</sup> min <sup>μ</sup><sup>m</sup>þ<sup>1</sup>

;

(Eq. (29)) to the desired fuzzy set of the software reliability levels:

:

� , <sup>μ</sup><sup>i</sup>

={0,1} is the characteristic function defining the presence of revision xi.

� � � � representing a set of ordered couples of revisions х<sup>i</sup> of the

1 ;…; μ<sup>j</sup>

n o

<sup>M</sup> <sup>¼</sup> <sup>m</sup>; <sup>μ</sup>Mð Þ <sup>m</sup> � � � � , (29)

ð Þ n�m

: (30)

� , <sup>μ</sup>mÞ: The number of relevant

mi, (32)

� �: (33)

� � � � , (34)

Þ, m ¼ 0, n; and Pm —reliability level defined according to the

(31)

Let us formalize the probable fuzziness of the software revision by transition from the charac-

universal set X и membership functions that characterize availability of revisions.

where μMð Þ m is the membership function defining the level of confidence in the fact that the

For the purpose of practical calculation, it is convenient to expand the revision membership

<sup>μ</sup><sup>0</sup> <sup>≥</sup> <sup>μ</sup><sup>1</sup> <sup>≥</sup> …<sup>≥</sup> <sup>μ</sup><sup>m</sup> <sup>≥</sup> <sup>μ</sup><sup>m</sup>þ<sup>1</sup> <sup>≥</sup>… <sup>≥</sup> <sup>μ</sup>n;

<sup>μ</sup><sup>0</sup> <sup>≤</sup> <sup>μ</sup><sup>1</sup> <sup>≤</sup> …<sup>≤</sup> <sup>μ</sup><sup>m</sup> <sup>≤</sup> <sup>μ</sup><sup>m</sup>þ<sup>1</sup> <sup>≤</sup> …<sup>≤</sup> <sup>μ</sup><sup>n</sup> :

revisions corresponding to the maximum level of confidence (i.e., to the maximum member-

<sup>m</sup> <sup>¼</sup> <sup>X</sup><sup>n</sup> i¼0

The maximum membership function can be calculated in the following way:

μmax ¼ min max

<sup>1</sup><i<<sup>m</sup> <sup>μ</sup><sup>i</sup>

By applying the generalization principle, we can move from the fuzzy set of relevant revisions

P ¼ Pm; μPð Þ Pm

; μi

In general, the membership function can be found using the following expression:

μ<sup>M</sup> ¼ maxmin μ<sup>i</sup><sup>1</sup> ; …μim ; μ<sup>j</sup>

Then, by defining the value α based on the specific software operating conditions and accuracy of expert estimation, we can obtain the interval (guaranteed) software reliability level:

$$P = \left\{ P\_m \, \vert \, \mu\_M(m) \ge \alpha \right\}. \tag{36}$$

#### 4.2. Example of possible application of fuzzy sets

Below is the simplest example of calculation of the software reliability level. During the debugging stage, 48 tests were carried out, 5 groups of defects were detected, and required revisions were performed. After the expert opinions were processed, the information on debugging was obtained in the form of a fuzzy set of revisions:

$$\mathbf{A} = \{ (1, 0.0), (2, 0.4), (3, 0.2), (4, 1.0), (5, 0.9) \}. \tag{37}$$

Having arranged the fuzzy set A by the membership function values, we obtained a fuzzy set of relevant revisions:

$$\mathbf{M} = \{ (0, 0.0), (1, 0.2), (2, 0.4), (3, 0.6), (4, 0.1), (5, 0.0) \}. \tag{38}$$

After we calculated reliability levels using the formulae (Eq. (3)), we obtained a fuzzy subset of the software reliability levels:

$$P = \{ (0.31, 0.2), (0.69, 0.4), (0.97, 0.6), (0.98, 0.1) \}. \tag{39}$$

According to the accepted assurance level α=0.4, we have.

$$\mathbf{P} = [0.69, 0.97]. \tag{40}$$

Thus, practical solutions suggested in the work take into account the uncertainties of software development and testing conditions. This allows obtaining rather accurate maximum and interval estimates of the software reliability and security. Analytical expressions allow simplifying the software reliability analysis as compared with the methods based on expert judgments. It is reasonable to apply the described results for planning of system and complex tests.

#### 5. Evaluation models and test planning selection criteria

It should be noted that there is no universal model of the software evaluation and test planning. Moreover, beside the described classes of models, studies suggest simulation models [29], structural models [22], fuzzy models [26, 27], interval models [30], software dynamic models [31–33], software/hardware complex models [34, 35], Bayesian model modifications [19, 30, 36, 37], as well as neural networks applied for certain scientific purposes [38, 39]. In order to select a suitable model, a number of qualitative and quantitative criteria can be suggested [40].

It is important to bear in mind that due to the dynamic nature, complexity, and heterogeneity of modern software development projects, the described models are not able to meet strict requirements for accuracy and serve for making intuitive decisions relating to the software test planning for all sets of input data. However, the results obtained from the model application are useful both for substantiating the labor content of the tests and for preparation of reports,

Models for Testing Modifiable Systems http://dx.doi.org/10.5772/intechopen.75126 165

1. The chapter presents a new class of probabilistic step models for software reliability (and security) assessment which allows to improve the adequacy and accuracy of evaluation for modern multi-version software systems (e.g., open-source software). One of the main features of the developed models is taking into account the effect of reducing the degree

These mathematical models have undergone a detailed study and lead to a method that allows planning and monitoring the level of software reliability at the stages of preliminary testing, trial operation, acceptance testing, inspection, and testing after modifications. Completeness and consistency of the method is ensured by the fact that the developed models do not impose strict limitations on the taxonomy of errors, modifications, tests,

2. The results of the proposed version of the test process modeling can be used at different stages of the software life cycle and integrated into various systems for modeling the reliability and safety of software. To do this the chapter proposes qualitative and quantita-

3. It should be mentioned that in the field of information security the use of mathematical models becomes a mandatory procedure in case of checking the high confidence level of the software. This is determined by the methodology of Common Criteria<sup>5</sup> regulated by

In the field of quality and functional safety of software, the application of mathematical models is welcomed to reduce the level of subjectivity in testing using black box method, fuzzing, functional testing, etc. (see the lines of international standards IEC 61508, IEC 61511, and ISO/ IEC 33001 and also the Russian new standard GOST R 56939). In this respect, IEC 61508– 7:2010<sup>6</sup> is extremely useful because it regulates the relationship between the classes of software

IEC 61508–7:2010 Functional safety of electrical/electronic/programmable electronic safety-related systems—Part 7:

which can increase the customer's confidence in the work deliverables.

of reliability when updating programs.

tive criteria for selecting software test models.

testing and the use of formal and semiformal models in detail.

6. Conclusion

and input data.

ISO/IEC 15408.

www.commoncriteriaportal.org

Overview of techniques and measures.

5

6

The following qualitative criteria can be used:


The following quantitative criteria can be used:


Combined and integral parameters, for instance:

$$I\mathbb{C} = \max \sum\_{i=1}^{K} k\_i \chi\_{i\prime} \tag{41}$$

where ki is the weighting factor of i property of the considered model selected by the expert and χi. is the characteristic function of the i property.

As the study has shown, there are a lot of mathematical models that allow estimating the software reliability and security at different stages of lifecycle, which is important for budget planning. On a practical level, the described classification of models simplifies selection and integration of the models based on the available statistics.

It is important to bear in mind that due to the dynamic nature, complexity, and heterogeneity of modern software development projects, the described models are not able to meet strict requirements for accuracy and serve for making intuitive decisions relating to the software test planning for all sets of input data. However, the results obtained from the model application are useful both for substantiating the labor content of the tests and for preparation of reports, which can increase the customer's confidence in the work deliverables.
