**4.3 Key performance indicators**

*Issues on Risk Analysis for Critical Infrastructure Protection*

The substeps that are performed comprise:

cyber threats.

1.Identify and prioritize assets: A list of identified assets indicating the

2.Identify threats: A threat is anything that could exploit a vulnerability to

importance of each one should be identified (e.g. software, hardware, data, interfaces, security governance, security controls and components, etc.).

breach security and cause harm to a CI. General threat categories are: physical adversarial threats and acts of terrorism, political/geopolitical/social threats, natural hazards, technological and accidental hazards, indirect threats and

3.Identify Vulnerabilities: Identify a list of known vulnerabilities of all the asset list and analyze the impact on the system/infrastructure if these are not correctly treated and mitigated The impact on the system shall be treated in terms

4.Analyze measures: Analyze the measures that are either in place or in the planning stage to minimize or eliminate the probability that a threat will

5.Determine the likelihood of an incident: The possibility of an incident to be an exploited vulnerability should be quantified, based on historical/statistical data, user experience and knowledge or any other sources available (e.g. studies, estimations/information that authorities are producing, etc.).

6.Assess the impact a threat could have, including factors such as the mission,

7.Prioritize the security risk: For each threat/vulnerability pair, determine the level of risk for the system/infrastructure, based on the likelihood and the impact of the threat, and the adequacy of the existing or planned system/ infrastructure security controls for eliminating or reducing the risk

8.Recommend Controls: Using the risk level from the previous step, determine the actions that the senior management of the CI and other personnel that hold key positions, must take to mitigate the risk to an accepted residual risk level.

9.Document the results to support management in making appropriate decisions

Following the above assessment, another process that can be used as a tool for identifying vulnerabilities and assessing performance is Penetration Testing (PT). PT is a security testing process in which experts execute real but yet controlled attacks on systems and services to identify methods for circumventing the security

1.A planning phase, focuses on gathering available information on the target systems, as well as on potential methods of attacks, management approval and setting the groundwork for setting up attack strategies and attack scenarios.;

the criticality and the sensitivity of the system and its data

on budget, policies, procedures, and so on.

features of an application, system, or network [44].

PT methodologies divide the process into four generic phases:

**4.2 Penetration testing**

of e.g. economy, reputation, and security for people

exploit a vulnerability in the system

**138**

KPIs typically enable the realization of technical systems towards tangible goals while serving as a benchmark for internal quality assurance. Indeed, KPIs are deemed as a measurable way to assess project's efficiency in reaching its key objectives and to evaluate the quality of the proposed technical solution(s). Through well-defined KPIs, the main areas to be tested, measured and validated during the piloting activities are established.

The SecureGas KPIs were defined in the early stage of the project so that they guide its targeted implementation. Preliminary activities, regarding user and system requirements identification as well as the CONOPS and HLRA definition, have already been completed providing valuable input to the KPIs definition task.

For the purposes of the SecureGas project, the KPIs were classified along two main indicator types:


Both the SecureGas component KPIs and the SecureGas Cross-KPIs establish the validation criteria to be measured during SecureGas pilot demonstrations. Although both KPI categories are equally important for the evaluation of objectives' fulfillment, this section emphasizes on the KPIs defined for the integrated SecureGas system (i.e. SecureGas Cross-KPIs).

The methodology adopted for the definition of the KPIs was built on a bottomup rationale. The SecureGas component KPIs (low level KPIs) were initially defined. Then, drawing on that information, the SecureGas Cross-KPIs (high level KPIs) were derived. The procedural pathway followed for the identification of KPIs is depicted in **Figure 3**.

Considering that KPIs depend on the end-users and stakeholders interested in the SecureGas system, the first step of the adopted methodology regarded their active engagement in the KPIs definition activities. This initiative had already started taking place through the definition of the user requirements (i.e. end-users needs and expectations from an integrated security system (such as the SecureGas system), as well as through dedicated stakeholders' workshops organized for the user requirements validation. The user requirements together with their external validation results shed light to those characteristics of the system that are deemed important by the end-users. In addition, information on the KPIs already applied by the end-users to assess the performance of their gas network daily operations allowed consortium partners to draft broad areas in which evaluations are

**Figure 3.** *KPIs definition pathway.*

performed. This information also enabled the consortium to examine how the SecureGas solution could contribute and add value to the resilience of end-users' infrastructure.

In parallel, drawing on the already defined technical requirements of the SecureGas components, consortium technical partners defined the key capabilities, characteristics and functionalities offered by every technical subsystem. The so-called SecureGas component KPIs enable components' development and implementation.

The next step regarded the definition of the SecureGas Cross-KPIs which reflect the most important features and characteristics offered by the entire (i.e. all subsystems integrated into one system) SecureGas solution. The end-users KPIs, the SecureGas component KPIs and the already defined SecureGas system specifications (Cross-Requirements), provided the baseline for the extraction of a list of eleven SecureGas Cross-KPIs (**Table 1**) that are key to performance success.

As presented in **Table 1**, the SecureGas Cross-KPIs were classified into specific Fields that outline the general domain categories where the impacts are going to exert their effect. Those Fields are as follows:


**141**

**Table 1.**

*SecureGas cross-KPIs.*

*Validation Strategy as a Part of the European Gas Network Protection*

Reliability False alert rate Percentage of false alerts (both

Cross correlation

Mean time to notify

> categories addressed

Automatic detection of threats

Automatic decisionsupport

integration of users' legacy systems

interface

capabilities (system health check)

Accuracy degradation percentage of a measurement value

Autonomy Threat

Interoperability Transparent

Usability Multilingual

Resilience Self-testing

**Field Indicator Description Metric Target value**

positive and negative) raised by the SecureGas system.

Percentage of cross correlated alerts raised by the SecureGas system.

moment an incident occurs and the moment the alert is displayed in the operational picture.

Time needed for the operator to create an incident notification and send it to competent authorities/ stakeholders (escalation of incident).

Number of different threats categories addressed by the SecureGas system (Threat categories: cyber, physical, cyber-physical, physical-cyber)

Number of different threat types automatically detected by the system. (Threat types: Intrusion detection, Third-Party Interference, Leak, Landslide hazard, Cyber)

Percentage of alerts automatically linked to recommendations on crisis management and mitigation actions

Number of users' legacy systems that can be easily and transparently integrated into the SecureGas system.

Number of different languages in which the SecureGas user interface will be available

Percentage of components/ sensors that provide information to the operator through dedicated alerts - about their status (not functioning and/or no communication)

The maximum decrease of accuracy (due to concept drift), before the model is retrained to adapt to background changes

Latency Time elapsed between the

% (False alerts / Total alerts)

% (Cross correlated alerts / Total alerts)

Time (sec) < 10 sec

Time (min) < 3 min

Number 4

Number ≥5

Number ≥1

Number 4 (English,

% 90–95%

% 20%

≥ 80

Italian, Greek, Lithuanian)

% (Alerts with decision support / Total alerts)

< 5%

> 50%

*DOI: http://dx.doi.org/10.5772/intechopen.94644*

• Interoperability, i.e. the ability of the system to work with new products (i.e. sensors or sub-systems) without special configurations.


*Validation Strategy as a Part of the European Gas Network Protection DOI: http://dx.doi.org/10.5772/intechopen.94644*

*Issues on Risk Analysis for Critical Infrastructure Protection*

performed. This information also enabled the consortium to examine how the SecureGas solution could contribute and add value to the resilience of end-users'

In parallel, drawing on the already defined technical requirements of the SecureGas components, consortium technical partners defined the key capabilities, characteristics and functionalities offered by every technical subsystem. The so-called SecureGas component KPIs enable components' development and

The next step regarded the definition of the SecureGas Cross-KPIs which reflect the most important features and characteristics offered by the entire (i.e. all subsystems integrated into one system) SecureGas solution. The end-users KPIs, the SecureGas component KPIs and the already defined SecureGas system specifications (Cross-Requirements), provided the baseline for the extraction of a list of eleven SecureGas Cross-KPIs (**Table 1**) that are key to performance

As presented in **Table 1**, the SecureGas Cross-KPIs were classified into specific Fields that outline the general domain categories where the impacts are going to

• Reliability, i.e. the capability of the system to function in a correct manner within the given timeframe. This includes high accuracy of alert localization, avoidance of any delays in data provision, and a low rate of false alerts or errors.

• Interoperability, i.e. the ability of the system to work with new products

(i.e. sensors or sub-systems) without special configurations.

• Autonomy, i.e. the level of independence of the system. An autonomous system is capable to operate (detect and process incidents) without human supervision

**140**

infrastructure.

*KPIs definition pathway.*

**Figure 3.**

implementation.

exert their effect. Those Fields are as follows:

(human in the loop only when deemed necessary).

success.

**Table 1.** *SecureGas cross-KPIs.*


Each of the aforementioned Fields was linked to a set of Indicators, each one being assigned a Description, Metric and Target Value.

Following the main principles of the SecureGas project, the SecureGas Cross-KPIs aimed and achieved to addresses all the Risk and Resilience phases. Those phases reflect the activities that need to be conducted before, during and after disruptive events, as part of a comprehensive risk and resilience management procedure. The Risk and Resilience phases are as follows: Prepare, Detect, Prevent, Absorb, Respond, Recover, Learn and Adapt. The ultimate goal of developing Cross-KPIs for all those phases was to showcase how the core functionalities and performance indicators of the SecureGas system can add value to the enhancement of the resilience of gas critical infrastructure networks. **Figure 4** presents the Risk and Resilience phases that are affected by each SecureGas Cross-KPIs. Some of the Cross-KPIs are linked to one phase, some others to more, while the Cross-KPI "Multilingual Interface" is related to all the seven Risk and Resilience phases, since the enhancement of the usability parameters of a system has the potential to affect the entire security and resilience status of a CI network.

**143**

components.

*Validation Strategy as a Part of the European Gas Network Protection*

**Figure 5** shows the KPIs distribution to the activities taking place before, during and after incidents. In general, the SecureGas Cross-KPIs are mostly linked to the activities/phases taking place before the occurrence of an incident (prepare, detect, prevent) (approx. 47.1% of KPIs), although the SecureGas system do have performance parameters that are related to the post incident activities (response, recover,

Within the context of the evaluation of SecureGas components and solution,

Regarding the first one, two types of questionnaires will be used for the evaluation purposes, one more generic that can be distributed to all participants (during testing, demonstrations, workshops) and one more specific, that would be filled by

1.Questionnaire 1 (generic): This will be addressed to all participants of the BC demonstrations and is based on the System Usability Scale (SUS), developed by John Brooke in 1986 [45]. The questionnaire 1 provides a "quick and dirty" though reliable tool for measuring the usability of tested systems. SUS consists of a 10-item questionnaire with five response options for respondents; from strongly agree to strongly disagree. This allows to gather evaluation feedback concerning a wide variety of products, systems and services, including hardware, software, mobile devices, websites and applications. SUS has become an

industry standard, with references in several articles and publications.

2.Questionnaire 2 (specific): The second questionnaire aims to extract endusers' assessed indicators on the basis of intuitiveness, usability, performance, etc. of the proposed solution. The end-users are going to fill-in this specific questionnaire after they have experienced the capabilities and the use of the system during the BC demonstration. This questionnaire is divided in seven main sections (i.e. general information, ease of installation, facilitation of user learning, data requirements, integrity, usability, usefulness), each one aimed at examining a different aspect of the end-users' view on the SecureGas

two main instruments will be used: questionnaires and interviews.

*KPIs distribution to the activities taking place before, during and after an incident.*

targeted participants within the audience, as further described below:

*DOI: http://dx.doi.org/10.5772/intechopen.94644*

learn and adapt) (approx. 32.4%).

**Figure 5.**

**4.4 Questionnaires and interviews**

**Figure 4.** *Risk and resilience phases affected by each SecureGas cross KPI.*

*Validation Strategy as a Part of the European Gas Network Protection DOI: http://dx.doi.org/10.5772/intechopen.94644*

*Issues on Risk Analysis for Critical Infrastructure Protection*

adapt to the evolving circumstances.

being assigned a Description, Metric and Target Value.

the entire security and resilience status of a CI network.

*Risk and resilience phases affected by each SecureGas cross KPI.*

implied set of users.

• Usability, i.e. is a set of attributes covering the effort needed for using a solution, and on the individual assessment of the use of the solution, by a stated or

• Resilience, i.e. is the ability of the SecureGas system to adapt from a disruption. This means that the system is able to identify potentially disruptive events and

Each of the aforementioned Fields was linked to a set of Indicators, each one

Following the main principles of the SecureGas project, the SecureGas Cross-KPIs aimed and achieved to addresses all the Risk and Resilience phases. Those phases reflect the activities that need to be conducted before, during and after disruptive events, as part of a comprehensive risk and resilience management procedure. The Risk and Resilience phases are as follows: Prepare, Detect, Prevent, Absorb, Respond, Recover, Learn and Adapt. The ultimate goal of developing Cross-KPIs for all those phases was to showcase how the core functionalities and performance indicators of the SecureGas system can add value to the enhancement of the resilience of gas critical infrastructure networks. **Figure 4** presents the Risk and Resilience phases that are affected by each SecureGas Cross-KPIs. Some of the Cross-KPIs are linked to one phase, some others to more, while the Cross-KPI "Multilingual Interface" is related to all the seven Risk and Resilience phases, since the enhancement of the usability parameters of a system has the potential to affect

**142**

**Figure 4.**

**Figure 5.** *KPIs distribution to the activities taking place before, during and after an incident.*

**Figure 5** shows the KPIs distribution to the activities taking place before, during and after incidents. In general, the SecureGas Cross-KPIs are mostly linked to the activities/phases taking place before the occurrence of an incident (prepare, detect, prevent) (approx. 47.1% of KPIs), although the SecureGas system do have performance parameters that are related to the post incident activities (response, recover, learn and adapt) (approx. 32.4%).
