**Meet the editor**

Leo D. Kounis is a Quality and Manufacturing Systems Engineer and Departmental Manager at the Manufacturing Directorate of the Hellenic Air Force's State Aircraft Factory, KEA, and is acting as Scientific Advisor to the Hellenic National Defence College.

He obtained his BEng (Hons) degree in Manufacturing Systems Engineering, his MSc degree in Quality Engi-

neering and his PhD degree in Systems Reliability from the University of Hertfordshire, United Kingdom. Dr. Kounis has worked as a Senior Quality Engineer in a number of private companies in Greece and has acted as a part-time lecturer at a number of universities.

His research interests focus on the area of quality, transportation and sustainable energy. He has published a number of scientific papers.

## Contents

## **Preface XI**


## Chapter 7 **ALAMEDA Ecosystem: Centering Efforts in Software Testing Development 155** José González Enríquez, Julián Alberto García-García, Francisco José Domínguez-Mayo and María José Escalona Cuaresma

#### Chapter 8 **Improving Quality Assurance in Multidisciplinary Engineering Environments with Semantic Technologies 177** Dietmar Winkler, Marta Sabou and Stefan Biffl

## Chapter 9 **The Use of Control Charts in the Study of Bitcoin's Price Variability 201** Beata Szetela

## Preface

Chapter 7 **ALAMEDA Ecosystem: Centering Efforts in Software Testing**

Domínguez-Mayo and María José Escalona Cuaresma

Chapter 8 **Improving Quality Assurance in Multidisciplinary Engineering Environments with Semantic Technologies 177** Dietmar Winkler, Marta Sabou and Stefan Biffl

Chapter 9 **The Use of Control Charts in the Study of Bitcoin's Price**

José González Enríquez, Julián Alberto García-García, Francisco José

**Development 155**

**VI** Contents

**Variability 201** Beata Szetela

This book on *Quality Control and Assurance* showcases the evolution of quality concepts, techni‐ ques and tools that are implemented by manufacturing companies and service providers.

The term *quality* stems from the Latin *qualitas*, which was introduced by Marcus Tullius Cic‐ ero, who created a Latin philosophical vocabulary, combining the Platonic method and the Roman philosophy. His influence to Latin and European languages is truly immense. Cicero was inspired by Plato's Academy, and it is indeed Plato and Aristotle who use the term *poiotes* originating from *poios.*

Aristotle, in his work *Categories,* discusses various forms of *qualities.* It is worth noting at this point that *quality* was defined by philosophers as an attribute, or characteristic of an object, so as to distinguish between certain levels and kinds of associated (given) or observed qualities.

Scientific findings show that Ancient Greeks did utilize qualitative control and non-destruc‐ tive testing (NDT) so as to pursue *good* quality. This included—but was not limited to—the manufacture of bronze fittings (concerning the erection of columns, such as the Philonian Stoa), to quality control of gold and silver coins.

The term *quality* begins to evolve in the science that is known today, when Walter Shewhart developed—in 1924—the notorious control chart. W. E. Deming, J. Juran and A. Feigen‐ baum, among others, further elaborated on the work of W. Shewhart and set the founda‐ tions of *total quality management (TQM)*.

Japan is regarded as the forerunner to establishing the framework of company-wide quality control, so as to increase its market share in the West. Scholars and business insiders alike acknowledge its influence. In order to counterbalance its momentum, the British Standard BS 5750 was developed, aiming at improving quality practices in an increasingly antagonis‐ tic environment. The advent of the International Standards Organization (ISO) 9000 became the internationally recognized quality management system standard. So much so that *Chrys‐ ler, Ford and General Motors*—although possessing their own quality manuals—oriented their efforts in accordance to the ISO 9000 series of standards, which formed the ISO/TS 16949 series of quality standards.

The *Malcolm Baldrige National Quality Award (MBNQA)* is only a handful of awards that pro‐ motes company excellence and successful performance. It goes without saying that the word *excellence* was soon adopted by enterprises and state legal entities, such as the *European Foundation for Quality Management (EFQM)* and the *British Quality Foundation (BQF)*, to en‐ hance qualitative practices. As such, it can be said that quality forms a holistic model that finds widespread application.

This book comprises nine chapters that cover a diverse area and is split in two sections.

The first section may be viewed as an introductory part that elaborates on the evolution of quality concepts and assurance by using statistical tools and techniques. In doing so, it presents a model that aims to capture customer's emotional responses in opting for a specif‐ ic product or service, which, in turn, may increase companies' market influence.

The ISO/IEC 17025 is the main ISO standard used by testing and calibration laboratories and is as such more explicit regarding competence-specific requirements. The presented paper dis‐ cusses key objects and strategies for laboratories to effectively overcome potential bottlenecks.

Youden plots are a simple graphical technique to analyze inter-laboratory results and ac‐ knowledge variations. The next study illustrates a method that is applied to determine con‐ taminants in surface waters and elaborates on findings in chemistry.

Lot quality assurance sampling (LQAS) is a random sampling technique first developed in 1920, aimed for use in manufacturing. However, LQAS has become an accepted sampling method in the area of public health that is recommended by the World Health Organization (WHO). Among others, it is used to monitor the prevalence of abortions and the quality of reproductive health care in Armenia.

The second section discusses quality frameworks and management in a number of enter‐ prises. TQM, although providing a platform for companies to improve their performance, was focused on the private sector and did not acknowledge the management of the life cycle of large infrastructure portfolios.

In order to bridge the gap, the Andalusian Public Administration in Spain called for the THOT project, so as to study in detail enterprise content management systems (ECMS). In doing so, a quality evaluation framework (QuEF) is proposed, intended to assist organiza‐ tions in selecting the most suitable ECM system.

The relationship between supply chain risk management (SCRM) and quality management is assessed in the next paper, and a link between TQM and SCRM schemes is identified.

The advent of information technology (IT) and the complexity of modern enterprises operat‐ ing in a globalized economy necessitated the interaction between technology and quality. This resulted in the development of a number of quality management software packages, such as enterprise resource planning (ERP), supply chain management (SCM) and customer relationship management (CRM), to name a few. Enterprise quality management software (EQMS) is a software platform that establishes cross-functional communication whilst standardizing, centralizing and streamlining management-specific data from across the val‐ ue chain. Preconfigured and industry-specific IT solutions are available that differ in design, functionality and capabilities.

To this end, the ISO/IEC/IEEE 29119 software testing standards intend to support a variety of software-specific life cycles and methods. The process per se is risk-based testing. It facili‐ tates the development of a strategy and supports test planning.

Indeed, model-driven engineering (MDE) is yet another software tool that concentrates on further developing conceptual models and is based on updated company-specific best prac‐ tices. It aims at optimizing in-between system compatibility whilst facilitating communica‐ tion between managers, designers and users of the application. In this context, established quality assurance (QA) methods need to be adapted to MDE so as to eliminate—among oth‐ ers—potential bottlenecks between incompatible data models.

In view of the above, the ALAMEDA software package provides support to life cycles fo‐ cused on the generation, implementation and testing from its initial phases. In doing so, it establishes a degree of compliance on par with the ISO/IEC-29119 standard. The outcome thereof—in conjunction with the implication of MDE—shows an impact in reducing time-tomarket and improved software-specific quality.

Semantic Web technologies (SWT) promote common data formats and exchange protocols on the web. They aim at bridging the gap between heterogeneous data models and sources and set the platform for establishing high levels of quality assurance. As such, the next pa‐ per introduces a set of success-critical requirements aimed at efficiently detecting shortcom‐ ings in MDE-governed environments whilst proposing a defect detection framework (DDF).

Virtual currencies are digital money that is not regulated and is controlled by its developers and circulated among users of a virtual community. The Bitcoin is the forerunner thereof and gained in popularity. The final chapter researches its volatility by comparing it to the US dollar (USD) and the Euro. This is done by means of control charts.

This book features a number of selected topics showcasing current implications of quality control and assurance.

It is hoped that it will serve as the basis for further research.

The first section may be viewed as an introductory part that elaborates on the evolution of quality concepts and assurance by using statistical tools and techniques. In doing so, it presents a model that aims to capture customer's emotional responses in opting for a specif‐

The ISO/IEC 17025 is the main ISO standard used by testing and calibration laboratories and is as such more explicit regarding competence-specific requirements. The presented paper dis‐ cusses key objects and strategies for laboratories to effectively overcome potential bottlenecks. Youden plots are a simple graphical technique to analyze inter-laboratory results and ac‐ knowledge variations. The next study illustrates a method that is applied to determine con‐

Lot quality assurance sampling (LQAS) is a random sampling technique first developed in 1920, aimed for use in manufacturing. However, LQAS has become an accepted sampling method in the area of public health that is recommended by the World Health Organization (WHO). Among others, it is used to monitor the prevalence of abortions and the quality of

The second section discusses quality frameworks and management in a number of enter‐ prises. TQM, although providing a platform for companies to improve their performance, was focused on the private sector and did not acknowledge the management of the life cycle

In order to bridge the gap, the Andalusian Public Administration in Spain called for the THOT project, so as to study in detail enterprise content management systems (ECMS). In doing so, a quality evaluation framework (QuEF) is proposed, intended to assist organiza‐

The relationship between supply chain risk management (SCRM) and quality management is assessed in the next paper, and a link between TQM and SCRM schemes is identified.

The advent of information technology (IT) and the complexity of modern enterprises operat‐ ing in a globalized economy necessitated the interaction between technology and quality. This resulted in the development of a number of quality management software packages, such as enterprise resource planning (ERP), supply chain management (SCM) and customer relationship management (CRM), to name a few. Enterprise quality management software (EQMS) is a software platform that establishes cross-functional communication whilst standardizing, centralizing and streamlining management-specific data from across the val‐ ue chain. Preconfigured and industry-specific IT solutions are available that differ in design,

To this end, the ISO/IEC/IEEE 29119 software testing standards intend to support a variety of software-specific life cycles and methods. The process per se is risk-based testing. It facili‐

Indeed, model-driven engineering (MDE) is yet another software tool that concentrates on further developing conceptual models and is based on updated company-specific best prac‐ tices. It aims at optimizing in-between system compatibility whilst facilitating communica‐ tion between managers, designers and users of the application. In this context, established quality assurance (QA) methods need to be adapted to MDE so as to eliminate—among oth‐

tates the development of a strategy and supports test planning.

ers—potential bottlenecks between incompatible data models.

ic product or service, which, in turn, may increase companies' market influence.

taminants in surface waters and elaborates on findings in chemistry.

reproductive health care in Armenia.

VIII Preface

of large infrastructure portfolios.

functionality and capabilities.

tions in selecting the most suitable ECM system.

**Dr. Leo D. Kounis** Hellenic Air Force State Aircraft Factory, KEA, Greece

**The Evolution of Quality and the Application of Quality Technique Tools**

**Provisional chapter**

## **The Evolution of Quality Concepts and the Related Quality Management The Evolution of Quality Concepts and the Related Quality Management**

Ching-Chow Yang Ching-Chow Yang Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/67211

#### **Abstract**

Enterprises usually adopt some quality practices to control the product quality during the manufacturing process in order to assure the delivery of qualitative good products to customers. The quality practices or quality management systems adopted by indus‐ tries will further evolve due to the changes of quality concepts as time goes by. This chapter discusses the change of quality concepts and the related revolution of quality management systems in the past century. The quality concepts were gradually changed from the achievement of quality standards, satisfaction of customer needs, and expecta‐ tions to customer delight. Since merely satisfying customers is not enough to ensure customer loyalty, the enterprises gradually focus on customers' emotional responses and their delight in order to pursue their loyalty. The emotion of "delight" is composed of "joy" and "surprise," which can be achieved as the customers' latent requirements are satisfied. Thus, the concept of "customer delight" and the means to provide the inno‐ vative quality so as to meet the unsatisfied customers' latent needs are elaborated on. Finally, a framework of innovation creation is developed that is based on the mining of customer's latent requirements. This outline will manifest the essential elements of the related operation steps.

**Keywords:** quality concept, customer satisfaction, customer delight, customers' latent requirement, innovation creation system

## **1. Introduction**

It is widely recognized that consumers only buy the goods with good quality, desired func‐ tions, and accepted price [1]. Therefore, industries always adopt several management prac‐ tices, or even develop some management systems to design and produce the products so as to meet customers' needs and expectations showcasing good quality and lower costs. But the

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

management practices or systems adopted by the industries are always changed due to the changes of the quality concepts as time goes by. This occurs as several quality gurus give the pragmatic definitions of "quality" over different time periods, thus causing industries to implement different practices or systems to control overall product quality [2].

In the first three decades of the last century, quality was defined as "conforming to the stan‐ dards and specifications of a product" [3]. Thus, the commonly adopted quality practices by industries were the standardization of quality, inspection, and rework. Deming empha‐ sized that "quality is to fulfill the requirements of customers and satisfy them" [4]. Hence, the meaning of quality was gradually changed to a "customer‐focused" perspective. Enterprises, therefore, committed themselves to satisfy customers' needs and expectations. Their aim was to pursue customer's satisfaction and loyalty [5, 6]. Companies also developed a number of methods to find out customers' needs and expectations. To this end, in‐depth interviews with customers were performed, customer surveys conducted, and market research was done. But, when Apple announced several innovative products, and their sales were increasing, it became apparent that only satisfying customers' requirements are not enough [7]. As a matter thereof the identification and fulfillment of customers' unsatisfied latent needs was gauged in conjunction with their emotional responses [8].

Entering the new century, several studies have suggested that merely satisfying customers is not enough to ensure their loyalty [9–11]. The industries need to focus on the customer's emotional responses and provide the products with attractive quality in order to pursue cus‐ tomers' delight [11]. It is worth mentioning at this point that Apple is regarded as a future trend setter, as it successfully created several innovative products such as iPod, iPhone, and iPad, triggering increased sales. A strong customer service department equally assisted in causing customers' delight experiences [12].

In view of the above, quality concepts have changed. Terms such as "customer delight" are deemed as forming the crucial elements of quality concepts, which are coined as "attractive quality" and "innovative quality" [13]. The conceptualization of delight is that the emotion of "delight" is composed of "joy" and "surprise," which can be achieved as the customers' latent requirements are satisfied. The provision of innovative quality products is the strategic tool to meet the unsatisfied customers' latent needs and their curiosity. These changes of quality concepts will lead the enterprises to reengineer their existing quality system, in order to develop the innovative quality attributes of products and services alike so as to retain and attract customers. This scenario causes the quality professionals and researchers to further develop and expand the new quality system beyond TQM. Based on performed research it is the author's view that research should focus on developing an effective new quality system.

During the past century, several quality gurus had led the changes of quality concepts. This chapter will discuss the changes of quality concepts during different time periods over the past century. To this end, it will also state the reflected revolutions of quality management systems with respect to the changing quality concepts. These are arranged in Sections 2 and 3, respectively. The development of "total quality management" (TQM) philosophy in the 1980s was an important landmark, which caused the change of new quality concepts and the reengineering of the quality management system, as introduced in Section 4. Based on the performance evaluation of the TQM implementation of TQM, it is worth noting at this point that the "business excellence model" had been proposed during the 1990s. This is elabo‐ rated in Section 5. This chapter also introduces the development of an integral model of a business excellence system beyond TQM, based on the realization of TQM.

Section 6 expands on the changes of new quality concepts pertaining to "customer delight" and "innovative quality" based on the investigation of customers' latent needs in the new century. This chapter includes a system, which may assist and cater for identifying and foster upon customers' undersatisfied latent needs and delights, enhancing innovative quality, which is addressed in Section 7. The conclusion of this chapter is listed in the final section.

## **2. The early quality concepts and the reflected quality management system**

Since the early twentieth century, manufacturing processes and activities initiated the con‐ trol practices to assure the product quality [14]. Manufacturing companies focused on the related productivity and manufacturing costs. As such, the quality concept and control were product‐focused.

#### **2.1. The quality concept of "standard" and the "inspection" control**

management practices or systems adopted by the industries are always changed due to the changes of the quality concepts as time goes by. This occurs as several quality gurus give the pragmatic definitions of "quality" over different time periods, thus causing industries to

In the first three decades of the last century, quality was defined as "conforming to the stan‐ dards and specifications of a product" [3]. Thus, the commonly adopted quality practices by industries were the standardization of quality, inspection, and rework. Deming empha‐ sized that "quality is to fulfill the requirements of customers and satisfy them" [4]. Hence, the meaning of quality was gradually changed to a "customer‐focused" perspective. Enterprises, therefore, committed themselves to satisfy customers' needs and expectations. Their aim was to pursue customer's satisfaction and loyalty [5, 6]. Companies also developed a number of methods to find out customers' needs and expectations. To this end, in‐depth interviews with customers were performed, customer surveys conducted, and market research was done. But, when Apple announced several innovative products, and their sales were increasing, it became apparent that only satisfying customers' requirements are not enough [7]. As a matter thereof the identification and fulfillment of customers' unsatisfied latent needs was

Entering the new century, several studies have suggested that merely satisfying customers is not enough to ensure their loyalty [9–11]. The industries need to focus on the customer's emotional responses and provide the products with attractive quality in order to pursue cus‐ tomers' delight [11]. It is worth mentioning at this point that Apple is regarded as a future trend setter, as it successfully created several innovative products such as iPod, iPhone, and iPad, triggering increased sales. A strong customer service department equally assisted in

In view of the above, quality concepts have changed. Terms such as "customer delight" are deemed as forming the crucial elements of quality concepts, which are coined as "attractive quality" and "innovative quality" [13]. The conceptualization of delight is that the emotion of "delight" is composed of "joy" and "surprise," which can be achieved as the customers' latent requirements are satisfied. The provision of innovative quality products is the strategic tool to meet the unsatisfied customers' latent needs and their curiosity. These changes of quality concepts will lead the enterprises to reengineer their existing quality system, in order to develop the innovative quality attributes of products and services alike so as to retain and attract customers. This scenario causes the quality professionals and researchers to further develop and expand the new quality system beyond TQM. Based on performed research it is the author's view that research should focus on developing an effective new quality system. During the past century, several quality gurus had led the changes of quality concepts. This chapter will discuss the changes of quality concepts during different time periods over the past century. To this end, it will also state the reflected revolutions of quality management systems with respect to the changing quality concepts. These are arranged in Sections 2 and 3, respectively. The development of "total quality management" (TQM) philosophy in the 1980s was an important landmark, which caused the change of new quality concepts and the reengineering of the quality management system, as introduced in Section 4. Based on

implement different practices or systems to control overall product quality [2].

gauged in conjunction with their emotional responses [8].

4 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

causing customers' delight experiences [12].

In 1913 Ford Motor Company, FMC, created the assembly line in their newly opened factory in Highland Park, Michigan due to the influence of the scientific management of Frederick W. Taylor. This resulted in Ford increasing its manufacturing volume [15]. Ford's assem‐ bly line was copied by many manufacturing companies. However, companies turned their attention to control product‐related quality issues. Since manufacturers were more "prod‐ uct‐focused" in that time, the quality concept was, therefore, aimed at "conforming to the standards and specifications of a product" [3]. This in turn, impelled quality engineers in manufacturing industries to implement the method of "inspection" so as to control the quality of a manufactured product.

Product designers and process engineers designed the standards and specifications of the products, which were based on their critical attributes. They also set up the standards of the manufacturing process and the standards of operations (SOPs). In doing so, the involved workforce was requested to perform the tasks according to the developed SOPs. The quality inspectors checked the dimensions and characteristics of products, detected the errors and failures, and took the necessary actions to improve the product quality.

#### **2.2. The development of process quality control**

In order to ensure product quality, companies needed to utilize the "full inspection" method. This was costly, since it required much time and labor efforts, and resulted in high internal quality costs [16]. Walter Shewhart, thus, created the control chart, a quality technique tool that he pioneered in Bell Laboratories working as a quality control engineer [17]. He proposed the use of a sampling inspection method instead of a 100% inspection to reduce the overall amount of inspection. The control chart was utilized to monitor the quality performance con‐ cerning the critical aspects of a process, whereas the attributes of the product were identified by means of sampling methods [18]. This enhanced overall effectiveness and also reduced the associated costs.

Sampling inspection and control charts use many statistic tools such as probability theory, the methods of random sample, analyses of sample mean and deviations to name but a few, as means for improving quality levels. Hence, the method of quality control suggested by Shewhart was called "statistical process control" (SPC) or "statistical quality control" (SQC). Sampling inspection may not always ensure product quality, as it might cause fewer defective products to be shipped to customers [19]. As a result, thereof, it leads to extra outside quality costs. Shewhart, however, had argued that if the missed number of defects was small, then the savings in inspection costs made it worthwhile [20].

## **3. The quality concepts and the reflected quality management system in the mid‐twentieth century**

More often than not, quality and the associated price of the goods are primary factors that are considered by customers prior to them materializing a purchase [21]. As such, manufacturing companies mainly emphasis on the control of quality and costs during the manufacturing process, especially the ones that trigger poor quality.

#### **3.1. The development of "quality costs"**

In essence, Juran propounded the concept of "quality costs" in his book "*Quality Control Handbook*" in 1951 [22]. Juran subdivided the quality costs into prevention, appraisal, internal failure, and external failure costs. The performed literature review indicates that the losses due to the production of defects and manufacturing failures are more than the costs of quality control; in particular, the costs caused by internal and external failures. In view of the above, the implementation of SPC could not effectively control the quality costs.

Despite this general classification being still widely used by practitioners and researchers, sev‐ eral authors had suggested other kinds of quality costs, especially the "invisible" or "hidden" costs. The term hidden cost is used to indicate failure costs that are inadequately recorded in company accounts and/or failure costs that are never actually discovered. Yang addressed the hidden costs through the definition and addition of two new categories: "extra resultant cost" and "estimated hidden cost" [16]. As such, the quality costs can be subdivided into the following six categories:


‐ external failure costs

amount of inspection. The control chart was utilized to monitor the quality performance con‐ cerning the critical aspects of a process, whereas the attributes of the product were identified by means of sampling methods [18]. This enhanced overall effectiveness and also reduced the

Sampling inspection and control charts use many statistic tools such as probability theory, the methods of random sample, analyses of sample mean and deviations to name but a few, as means for improving quality levels. Hence, the method of quality control suggested by Shewhart was called "statistical process control" (SPC) or "statistical quality control" (SQC). Sampling inspection may not always ensure product quality, as it might cause fewer defective products to be shipped to customers [19]. As a result, thereof, it leads to extra outside quality costs. Shewhart, however, had argued that if the missed number of defects was small, then the

**3. The quality concepts and the reflected quality management system in** 

More often than not, quality and the associated price of the goods are primary factors that are considered by customers prior to them materializing a purchase [21]. As such, manufacturing companies mainly emphasis on the control of quality and costs during the manufacturing

In essence, Juran propounded the concept of "quality costs" in his book "*Quality Control Handbook*" in 1951 [22]. Juran subdivided the quality costs into prevention, appraisal, internal failure, and external failure costs. The performed literature review indicates that the losses due to the production of defects and manufacturing failures are more than the costs of quality control; in particular, the costs caused by internal and external failures. In view of the above,

Despite this general classification being still widely used by practitioners and researchers, sev‐ eral authors had suggested other kinds of quality costs, especially the "invisible" or "hidden" costs. The term hidden cost is used to indicate failure costs that are inadequately recorded in company accounts and/or failure costs that are never actually discovered. Yang addressed the hidden costs through the definition and addition of two new categories: "extra resultant cost" and "estimated hidden cost" [16]. As such, the quality costs can be subdivided into the

the implementation of SPC could not effectively control the quality costs.

associated costs.

savings in inspection costs made it worthwhile [20].

6 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

process, especially the ones that trigger poor quality.

**3.1. The development of "quality costs"**

following six categories:

‐ internal failure costs

‐ prevention costs ‐ appraisal costs

**the mid‐twentieth century**


**Figure 1** depicts the new definition of quality costs.

**Figure 1.** Definition of categories of quality costs.

#### **3.2. The quality concept "quality assurance" and the related TQC‐system**

After Juran and several quality experts emphasized the quality cost issue, the concept of qual‐ ity costs was widely accepted by industry. Meanwhile, another quality concept appeared gradually, namely "quality assurance." However, a large number of experts in the area of quality control, including Feigenbaum, asserted that implementing an SPC system on its own could not effectively control quality costs [23]. It is worth mentioning that the concept of "quality assurance" was "users‐oriented," implying that "product possesses the fitness for purpose of use based on its functions" and hence "quality is zero defects and meeting the specifications 100%" [24].

The performed literature review suggests that Feigenbaum emphasized that quality assur‐ ance could not be achieved by placing the control just on production processes. Thus, the concept of total quality control, TQC, was introduced in 1959. TQC emphasizes that product quality needs to be implemented at all stages of the product life cycle. The sequence of quality activities is briefly shown as:


Actually, Feigenbaum's quality concept and ideas were similar to those described by Deming, Juran, and Crosby [25].

The concepts and approaches of SPC, TQC, and "costs of quality" were introduced in Japan during 1960 by Deming and Juran. The Union of Japanese Scientists and Engineers (JUSE), which was formed in 1946, synthesized the concepts, principles, and approaches of statistical process control and total quality control [26]. JUSE promoted the practices of TQC and the quality concepts pursuing the zero defect culture and executing the task right first time.

## **4. The age of total quality management**

While Japanese industries adopted the TQC practices, they emphasized the education and training of quality for all employees and the cultivation of a quality culture. Therefore, the implementation of TQC in Japanese industries was very different from the original TQC.

#### **4.1. The emerging of the Japanese company‐wide quality control, CWQC**

Actually, the Japanese TQC possessed several critical characteristics listed as follows:


The performed literature review suggests that Feigenbaum emphasized that quality assur‐ ance could not be achieved by placing the control just on production processes. Thus, the concept of total quality control, TQC, was introduced in 1959. TQC emphasizes that product quality needs to be implemented at all stages of the product life cycle. The sequence of quality

Actually, Feigenbaum's quality concept and ideas were similar to those described by Deming,

The concepts and approaches of SPC, TQC, and "costs of quality" were introduced in Japan during 1960 by Deming and Juran. The Union of Japanese Scientists and Engineers (JUSE), which was formed in 1946, synthesized the concepts, principles, and approaches of statistical process control and total quality control [26]. JUSE promoted the practices of TQC and the quality concepts pursuing the zero defect culture and executing the task right first time.

While Japanese industries adopted the TQC practices, they emphasized the education and training of quality for all employees and the cultivation of a quality culture. Therefore, the implementation of TQC in Japanese industries was very different from the original

**4.1. The emerging of the Japanese company‐wide quality control, CWQC**

○ customer‐focused and quality‐first as the quality policies,

○ education and training of quality for all employees,

○ realization of "do the right thing first time,"

Actually, the Japanese TQC possessed several critical characteristics listed as follows:

activities is briefly shown as:

8 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

‐ incoming quality approval,

‐ process quality control,

‐ product reliability,

‐ inventory,

TQC.

‐ delivery, and

‐ customer service.

Juran, and Crosby [25].

**4. The age of total quality management**

○ full participation and teamwork,

‐ product design,

Based on aforementioned critical characteristics, the Japanese TQC was acknowledged as company‐wide quality control (CWQC).

The implication of CWQC in conjunction with Japanese industrial competitiveness and stra‐ tegic advantages facilitated their entrance into western markets. Japanese enterprises enjoyed an increase in global market share by providing the customers with high‐quality products at lower prices [27]. This in turn, resulted in western companies facing an increased amount of competition from Japanese and other Asian manufacturers.

#### **4.2. The development of total quality management**

The Japanese competition caused American and western industries to implement bench‐ marking projections by studying Japanese CWQC performances. In doing so, they adopted the principles of Japanese quality management. Based on the critical characteristics listed above, western professionals further developed and refined CWQC to become a total quality management (TQM) system. Its fundamental principles are listed as follows:


It is the author's view that the TQM was well developed and suitable for western organiza‐ tions, as depicted in **Figure 2**. Thus, it was widely adopted by industries and nonprofit orga‐ nizations around the world.

**Figure 2.** The framework of TQM.

The development of TQM was also influenced by the western quality experts, namely Deming, Juran, and Crosby [28]. As already stated Deming's quality concept was that "qual‐ ity is to fulfill the requirements of customers and satisfy them" [4]. Crosby's definition of quality was also similar, as he defined quality as "conformance to customers' requirements." Juran also defined quality as being "fitness for purpose of use, …, it is judged by the users, not the manufacturers, or the merchants" [22]. TQM was thus an integrated model of man‐ agement philosophies, quality concepts, and a set of practices, which were also influenced by Deming's 14 points and Juran's quality trilogy. However, to implement the TQM successfully it is necessary to integrate the so‐called "hard side" elements (that is, statistical methods, quality control tools, process standardization, and improvement, etc.) with the "soft side" aspects (that is, quality concept, employees' participation, education and training, and quality culture, etc.) [29].

From the mid‐1980s onward, several important quality programs were being launched. Besides the development of TQM, the ISO 9000 system and the Six‐Sigma program (which was initiated by Motorola) commenced in 1987. It is worth noting that up to date, the ISO system has had four revisions in 1994, 2000, 2008, and 2015, respectively. The Six‐Sigma quality scheme was being widely imitated by GE in 1995 [30]. The successful implementation of Six‐Sigma by Motorola, GE, and several other multinational companies caused the Six‐Sigma philosophy to be globally adopted by a number of industries and organizations [31].

## **5. From Malcolm Baldrige National Quality Award to the Business Excellence Model**

From the late 1990s to present, the traditional quality concept has seen a number of changes. In the United States of America, the Department of Commerce introduced in 1987 the Malcolm Baldrige National Quality Award (MBNQA) by benchmarking the Japanese Deming Award [32] to promote the implementation of TQM in industries and nonprofit organizations.

#### **5.1. The national quality award and TQM**

The main aim of TQM implementation is to achieve customers' satisfaction [33]. This is turn results in a company improving its financial performances [34–36]. The benefits may be seen in areas such as cost reduction, increased market share and profit, and enhanced busi‐ ness competitiveness [37]. As a result, MBNQA specifically emphasizes business excellence. Besides the implementation of TQM, MBNQA also considers the strategic management, and information management and analyses.

The European Foundation for Quality Management (EFQA) also launched in 1992 a European Quality Award (EQA) [38]. Based on the performed research, more than 80 countries have established National Quality Awards (NQAs). These NQAs are largely based on the MBNQA, EQA, and the Deming Prize [39, 40], and are generally considered to be an effective way of pursuing business excellence, including customer loyalty and better long‐term financial per‐ formances. Most of these national quality awards may form part of the TQM system namely, strategic management, performance evaluation, human resource management, and even IT system and innovation.

#### **5.2. Prospects of the business excellence model beyond TQM**

The framework of the National Quality Award is indeed beyond the extent of TQM. In most of the cases, National Quality Award such as MBNQA and EQA are acknowledged as a busi‐ ness excellence model, with the industries adopting it as their strategic business system. Yang [41] proposed a framework of an integral model of a business‐excellence system based on the integration of TQM, MBNQA, and EQA, and its islands:


The development of TQM was also influenced by the western quality experts, namely Deming, Juran, and Crosby [28]. As already stated Deming's quality concept was that "qual‐ ity is to fulfill the requirements of customers and satisfy them" [4]. Crosby's definition of quality was also similar, as he defined quality as "conformance to customers' requirements." Juran also defined quality as being "fitness for purpose of use, …, it is judged by the users, not the manufacturers, or the merchants" [22]. TQM was thus an integrated model of man‐ agement philosophies, quality concepts, and a set of practices, which were also influenced by Deming's 14 points and Juran's quality trilogy. However, to implement the TQM successfully it is necessary to integrate the so‐called "hard side" elements (that is, statistical methods, quality control tools, process standardization, and improvement, etc.) with the "soft side" aspects (that is, quality concept, employees' participation, education and training, and quality

From the mid‐1980s onward, several important quality programs were being launched. Besides the development of TQM, the ISO 9000 system and the Six‐Sigma program (which was initiated by Motorola) commenced in 1987. It is worth noting that up to date, the ISO system has had four revisions in 1994, 2000, 2008, and 2015, respectively.

culture, etc.) [29].

**Figure 2.** The framework of TQM.

10 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**Figure 3.** The framework of the proposed integral model of a business‐excellence system.

Meanwhile, the concept of "sustainable development" was also widely adopted in industry [42, 43]. Zairi asserted that "sustainable development is based on a perceived need to address environmental deterioration and to maintain the vital functions of natural systems for the well‐being of present and future generations" [44]. He thus, proposed an integrated model that incorporates the sustainable development into a TQM system. Yang et al. developed a framework of the system of environmental management, which constitutes of 30 activity items themselves belonging to 12 initiatives [45].

## **6. The concepts of "customer delight" and "innovative quality" in the new century**

A critical strategy of pursuing business excellence is to raise customers' loyalty, which cannot be assured by grossly satisfying customers. Indeed, even satisfied customers would defect at a significant rate in many industries. It is recognized that customers' satisfaction is the result of the fulfillment of their explicit needs. As such, satisfaction only of their needs does not directly result in customer loyalty, since the competitors would also fulfill these explicit needs with better customer‐valued products [46].

Several studies have revealed that the degree of loyalty depends on whether the customers are "totally satisfied" or merely "satisfied" [47, 48]. The totally satisfied customers are actu‐ ally about five times more likely to repurchase a product or service than those who are merely "satisfied" [49, 50]. This suggests that enterprises must strive for "total customer satisfaction," or even "customer's delight" [51].

"Customer delight" is a new quality concept. A conceptualization of "delight" is the custom‐ er's emotional response, which is composed of "joy" and "surprise"—both encountered in the providing process of goods/services [52]. In order to delight customers, the enterprises need to provide the goods/services with attractive and innovative quality incentives. The strategic actions are to identify customers' latent needs and to create customer value by developing the innovative products and attractive services in order to fulfill these.

If a company brings out some innovative products that consumers did not know they needed before, then these innovative products will often spark high demand [53]. Indeed, Moore illustrated eight types of innovations, namely [54]:


Meanwhile, the concept of "sustainable development" was also widely adopted in industry [42, 43]. Zairi asserted that "sustainable development is based on a perceived need to address environmental deterioration and to maintain the vital functions of natural systems for the well‐being of present and future generations" [44]. He thus, proposed an integrated model that incorporates the sustainable development into a TQM system. Yang et al. developed a framework of the system of environmental management, which constitutes of 30 activity

**Figure 3.** The framework of the proposed integral model of a business‐excellence system.

**6. The concepts of "customer delight" and "innovative quality" in the** 

A critical strategy of pursuing business excellence is to raise customers' loyalty, which cannot be assured by grossly satisfying customers. Indeed, even satisfied customers would defect at

items themselves belonging to 12 initiatives [45].

12 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**new century**

These kinds of innovations, especially the first four types, will result in significant effects on the fulfilment of the customers' latent needs and their delight experiences.

## **7. The development of the innovation system**

In order to raise the customers' loyalty, the enterprises should develop the core capabilities with an innovation system. In view of the above, the prerequisite for developing such a sys‐ tem is been described. As such, this chapter proposes a systematic innovation development system based on the product/service value chain.

#### **7.1. The framework of the innovation system**

At first, the differences among improvement, reengineering, and innovation are established as shown in **Figure 4**. Improvement is implemented on the existing processes, as some prob‐ lems may have been encountered during the manufacturing process or the service delivery process. Continuous improvement is the key activity of the TQM system, its aim is to improve the quality of products/services and all the observed and or experienced bottlenecks on the processes. The final objective is to raise customer satisfaction [55].

**Figure 4.** The framework of the innovation system.

Hammer and Champy asserted that improving solely the production system or service deliv‐ ery processes may not result in significant business performance [56]. Sometimes it is needed to reengineer the critical processes, or to redevelop the provision systems based on customer voices. The latter applies in particular to key customer values which are difficult to fulfill by adopting the improvement actions, as shown in **Figure 4**. Thus, the aim of reengineering is to create customers' values, which are the critical factors of raising customer loyalty. Besides, reen‐ gineering also reaps the benefits including shorter delivery times of products/services (includ‐ ing lead time and production time), reduction of costs, and the effective utilization ofresources.

Most companies develop and implement enterprise resources planning (ERP), customer rela‐ tionship management (CRM), and supply chain management (SCM) simultaneously [57]. By integrating these systems, enterprises can exert excellent business' performance and customer's loyalty. In order to effectively utilize these networks and their integrated systems, it is the author's view that companies should proceed with the necessary processes of reengi‐ neering. This in turn, may have an impact on the critical functional activities on a timely basis.

The utilization of high‐tech functions and the rapid application of the internet resulted in the change of customers' purchasing behavior [58, 59]. Customers now focus not only on the eval‐ uations of price and functions; rather they also integrate quality with the perceived value [60]. It is the author's view that business systems ought to change and pursue innovation, speed, and quality if they want to satisfy the overall needs of customers and create new value.

Customer satisfaction may be termed as fulfilling their requirements and expectations. This, however, is not enough for delighting them and raising their loyalty. Performed research indi‐ cates that a number of customer's latent needs are never discovered and satisfied by enterprises [61]. Thus, the latter's competitive force is to probe the unmet customers' latent needs. There are some new methodologies proposed by several researchers to pinpoint customers' latent needs [62]. Once the latter are identified, companies ought to overcome bottlenecks and aim at develop‐ ing and providing innovative products/services to satisfy customers' latent needs and curiosity.

Eventually the customer will be set at the center of innovation. Thomke and von Hippel asserted that "tapping into customer innovation can certainly generate tremendous value, but capturing customer value is hardly a simple or straightforward work" [63]. The term "innova‐ tion" is usually considered as a process or a capability to create new ideas, and then develop the innovative products/services in order to meet customers' unsatisfied latent needs [64, 65]. In essence, the realization of "innovation" depends on the implementation of an innovation system, as shown in **Figure 4**. This innovation system starts at the "mind" mining of the cus‐ tomer's latent needs and ends in the customer's value and response, and vice versa.

#### **7.2. The transfer loop of innovation system**

**7.1. The framework of the innovation system**

14 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**Figure 4.** The framework of the innovation system.

processes. The final objective is to raise customer satisfaction [55].

At first, the differences among improvement, reengineering, and innovation are established as shown in **Figure 4**. Improvement is implemented on the existing processes, as some prob‐ lems may have been encountered during the manufacturing process or the service delivery process. Continuous improvement is the key activity of the TQM system, its aim is to improve the quality of products/services and all the observed and or experienced bottlenecks on the

Hammer and Champy asserted that improving solely the production system or service deliv‐ ery processes may not result in significant business performance [56]. Sometimes it is needed to reengineer the critical processes, or to redevelop the provision systems based on customer voices. The latter applies in particular to key customer values which are difficult to fulfill by adopting the improvement actions, as shown in **Figure 4**. Thus, the aim of reengineering is to create customers' values, which are the critical factors of raising customer loyalty. Besides, reen‐ gineering also reaps the benefits including shorter delivery times of products/services (includ‐ ing lead time and production time), reduction of costs, and the effective utilization ofresources. Most companies develop and implement enterprise resources planning (ERP), customer rela‐ tionship management (CRM), and supply chain management (SCM) simultaneously [57]. By integrating these systems, enterprises can exert excellent business' performance and customer's loyalty. In order to effectively utilize these networks and their integrated systems, it is the author's view that companies should proceed with the necessary processes of reengi‐ neering. This in turn, may have an impact on the critical functional activities on a timely basis.

In order to effectively implement the innovation system, a transfer loop of the innovation system has been developed, as shown in **Figure 5**. It constitutes of four constructs:

**Figure 5.** The transfer loop of the innovation system.


It is worth mentioning that each construct contains several items. As such, the construct "tech‐ nology innovation" is termed as the fundamental force of innovation, which contains the following items:


These are driven by the following core activities:


The effective implementation of "technology innovation" can create the "innovative prod‐ ucts," which includes the following:


Each of the other three constructs also contains several items. These constructs and their involving items are shown in **Figure 5**.

The features of "innovative products" will satisfy the "customer value" which is accentuated by good quality, delight, good experience, and extra value‐added features. In order for the companies to fulfill these characteristics, it is hereby suggested to use "mind mining" methods to identify customers' unsatisfied and latent needs. They also ought to use methods and means to eliminate the inconvenient utilization, and to integrate product functions and quality. These findings and results may be hereby included into the quality functions. For realizing these quality functions, the drivers are the capability of "technology innovation." As such, the four constructs form a "transfer loop of the innovation system," as shown in **Figure 5**.

## **8. Conclusion**

‐ technology innovation,

It is worth mentioning that each construct contains several items. As such, the construct "tech‐ nology innovation" is termed as the fundamental force of innovation, which contains the

The effective implementation of "technology innovation" can create the "innovative prod‐

Each of the other three constructs also contains several items. These constructs and their

The features of "innovative products" will satisfy the "customer value" which is accentuated by good quality, delight, good experience, and extra value‐added features. In order for the companies to fulfill these characteristics, it is hereby suggested to use "mind mining" methods

‐ innovative products, ‐ customer value, and ‐ product functions.

following items:

‐ insight,

‐ creativity,

‐ core capability,

‐ experience share,

‐ cooperation.

‐ resource inputs, and

‐ professional knowledge,

ucts," which includes the following:

‐ good quality at an affordable price.

involving items are shown in **Figure 5**.

‐ integrated multifunctions,

‐ curiosity satisfaction,

‐ easy to use, and

‐ technology development,

‐ product content, and ‐ innovation process.

‐ development of new material,

These are driven by the following core activities:

16 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

In a highly globalized economy, it is widely recognized that "innovation" has become the key force for achieving business excellence, which in turn reflects as competitive advantages, growth, and development [66]. Only the pursuit of product quality and service quality is not enough to achieve the business objectives. The enterprises should alter their quality concepts from the narrow definition of quality to include customer value and innovation. In order to realize the innovation performance, this study has proposed a framework of the innova‐ tion system and the related transfer loop of the system. However, the proposed system is a conceptual model, empirical studies ought to be conducted to identify potential causalities among the critical constructs and their practices.

## **Author details**

Ching‐Chow Yang

Address all correspondence to: chinchow@cycu.edu.tw

Department of Industrial and Systems Engineering, Chung Yuan Christian University, Taiwan, Republic of China

## **References**


[24] Crosby, P. *Quality is Free*. 1979; New York: McGraw‐Hill.

[7] Teece, D. J. Business model, business strategy and innovation. *Long Range Planning*, 2010;

[8] Rindova, V. P., & Petkova, A. P. When is a new thing a good thing? Technological change, product form design, and perceptions of value for product innovations. *Organization* 

[9] Reichheld, F. F. Loyalty and renaissance of marketing. *Marketing Management*, 1994; 2(4),

[10] Schneider, B., & Bowen, D. E. Understanding customer delight and outrage. *Sloan* 

[11] Kumar, A., Olshavsky, R. W., & King, M. F. Exploring alternative antecedents of customer delight. *Journal of Customer Satisfaction, Dissatisfaction and Complaining Behaviour*, 2001; 14,

[13] Yang, C.‐C. The identification of customer delight for quality attributes and its applica‐

[14] Rao, A., Carr, L. P., Dambolena, I., Kopp, R. J., Martin, J., Rafii, F., & Schlesinger, P. F. *Total Quality Management: A Cross Functional Perspective.* 1996; New York: John Wiley

[15] Zokaei, K., & Simons, D. Performance improvements through implementation of lean practices: A study of the U.K. Red Meat Industry. *International Food and Agribusiness* 

[16] Yang, C.‐C. Improving the definition and quantification of quality costs. *Total Quality* 

[17] Mitra A. *Fundamentals of Quality Control and Improvement Second edition.* 1998; London:

[18] Xie, M., Goh, T. N., & Ranjan, P. Some effective control chart procedures for reliability

[19] Lavin, M. Inspection efficiency and sampling inspection plans. *Journal of the American* 

[20] Giakatis, G., Enkawa, T. & Washitani, K. Hidden quality costs and the distinction between quality cost and quality loss, *Total Quality Management,* 2001; 12(2), 179–190.

[21] Oakland, J. S. *Total Quality Management.* 2nd ed. 2000; Jordan Hill, Oxford: Butter‐

[23] Feigenbaum, A. V. Total Quality Control. *Harvard Business Review*, 1956; 34 (6), 93–101.

monitoring. *Reliability Engineering & System Safety*, 2002; 77(2), 143–150.

[22] Juran, J. M. (1974).*Quality Control Handbook.* 3rd ed. New York: McGraw‐Hill.

tions. *Total Quality Management & Business Excellence*, 2011; 22(1–2), 83–98.

43, 172–194.

10–21.

14–26.

& Sons.

*Science*, 2007; 18(2), 217–232.

*Management Review*, 1999; 41(1), 35–45.

18 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

*Management Review*, 2006; 9(2), 30–53.

Prentice‐Hall International (UK) Limited.

*Statistical Association*, 1946; 41(236), 432–438.

worth‐Heinemann.

[12] Gallo, C. *The Apple Experience.* 2012; New York: McGraw‐Hill Inc.

*Management & Business Excellence*, 2008; 19(3–4), 175–191.


[41] Yang, C.‐C. Development of an integrated model of a business excellence system. *Total* 

[42] Kanji, G. K. Architecture of business excellence in the public and service sectors. *Total* 

[43] Okumura, S., Morikuni, T., & Okino, N. Environmental effects of physical life span of a reusable unit following and physical failures in a remanufacturing system. *International* 

[44] Zairi, M. Beyond TQM implementation: the new paradigm of TQM sustainability. *Total* 

[45] Yang, C.‐C. Exploration strategies and key activities for the system of environmental management. *Total Quality Management & Business Excellence,* 2011; 22(11), 1179–1194.

[46] Matzler, K., & Hinterhuber, H. H. How to make product development projects more successful by integrating Kano's model of customer satisfaction into quality function

[47] Finkelman, D. P., & Goland, A. R. How not to satisfy your customers. *The McKinsey* 

[48] Heskett, J. L., Jones, T.O., Loveman, G.W., Sasser, W. E., & Schlesinger, L.A. Putting the

[49] Jones, T. O., & Sasser Jr., W. E. Why satisfied customers defect. *Harvard Business Review*,

[50] Johnson, R., Retail isn't broken Stores are. *Harvard Business Review*, 2011; 89(12), 78–82. [51] Johann, F., & Kurt, M. Customer delight and market segmentation: An application of three‐factor theory of customer satisfaction on lift style groups. *Tourism Management*,

[52] Torres, E. N., & Shery, k. From customer satisfaction to customer delight: Creating a new standard of service for the hotel industry. *International Journal of Contemporary Hospitality* 

[53] Rigby, D. K., Gruver, K., & Allen, J. Innovation in turbulent times, *Harvard Business* 

[54] Moore, G. A., Innovating within established enterprises. *Harvard Business Review*, 2004;

[55] Yang, C.‐C., The establishment of a TQM system for the health care industry. *The TQM* 

[56] Hammer, M., & Champy, J. *Reengineering the Corporation.* 1993; New York: Harper Collins. [57] Hendricks, K. B., Singhai, V. R. & Stratman, J. K. The impact of enterprise systems on cor‐ porate performance: A study of ERP, SCM, and CRM system implementations. *Journal of* 

service‐profit chain to work. *Harvard Business Review*, 1994; 72(2), 164–174.

*Quality Management & Business Excellence*, 2009; 20(9), 931–944.

*Journal of Production Research,* 2003; 43(16), 3667–3687.

20 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

development. *Technovation*, 1998; 18(1), 25–38.

*Quarterly*, 1990; 26(4), 2–12.

1995; 73(6), 88–99.

2008; 29(1), 116–126.

*Management*, 2013; 25(5), 642–659.

*Review*, 2009; 87(6), 79–86.

*Magazine*, 2003; 15(2), 93–98.

*Operations Management*, 2007; 25(1), 65–82.

82(7–8), 87–92.

*Quality Management and Business Excellence*, 2008; 19(3–4), 399–415.

*Quality Management & Business Excellence*, 2002; 13(8), 1161–1172.


## **Key Aspects for Implementing ISO/IEC 17025 Quality Management Systems at Materials Science Laboratories Key Aspects for Implementing ISO/IEC 17025 Quality Management Systems at Materials Science Laboratories**

Rodrigo S. Neves, Daniel P. Da Silva, Carlos E. C. Galhardo, Erlon H. M. Ferreira, Rafael M. Trommer and Jailton C. Damasceno Carlos E. C. Galhardo, Erlon H. M. Ferreira, Rafael M. Trommer and Jailton C. Damasceno Additional information is available at the end of the chapter

Rodrigo S. Neves, Daniel P. Da Silva,

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66100

#### **Abstract**

Implementing a quality management system based on the requirements specified in ISO/IEC 17025 standard at materials science laboratories is challenging, mainly due to two main factors: (i) the high technical complexity degree of some tests used for materials characterization and (ii) the fact that most materials science laboratories provide materials characterization tests and also carry out research and development activities. In this context, this chapter presents key subjects while implementing a quality management system at materials science laboratories and some considerations on strategies for effectively implementing such systems.

**Keywords:** quality management, quality assurance, materials metrology, ISO/IEC 17025, materials science laboratories

### **1. Introduction**

The constant strive for innovation on the development of new products and services led, in the last decades, to a growing interaction between research and development (R&D) laboratories. Most of these laboratories are from universities and research institutes and interact with diverse branches of productive sector. As a consequence, several of them opted for implementing a quality management system based on an international requirement standard aiming at improving products and services reliability [1–6].

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Particularly interesting cases are the materials science laboratories [7]. Many of these labora‐ tories, originally headed almost exclusively to R&D activities, became interested in providing accredited tests for materials characterization. As a consequence, they should adopt a quality management system based on the ISO/IEC 17025 requirements standard [8]. On the other hand, materials characterization laboratories specialized in providing accredited high‐technology tests are constantly pushed to carry out R&D, in order to improve or develop new processes and services. One way or another, both cases describe scenarios where quality management system must run in an environment where R&D and accredited testing services co‐exist.

Considering this context, implementing quality management systems at materials science laboratories is a challenging task due to several reasons. The first point is the fact that whilst the implementation of an ISO/IEC 17025 oriented quality management system at testing laboratories is a consensus, the advantages and disadvantages of applying quality manage‐ ment concepts to R&D activities are still issues widely discussed by the quality management community. Indeed, the main aspects of this controversy are highlighted in the articles of Mathur‐De‐Vré [9] and Krapp [10]. They state that adding quality management concepts, like metrological traceability, trackability and rational human resources management, must not restrain the flexibility necessary in order to carry out R&D activities.

In spite of this contradiction, materials science laboratories, but also R&D laboratories of several scientific areas, start to see the standardization promoted by the implementation of a quality management system based on ISO/IEC 17025 standard as a way to reduce costs, reduce tests execution deadlines, satisfy customers and especially add quality to their services and products. More recently, an article published in *Nature* has drawn attention to how quality management and quality assurance concepts could improve R&D activities and results [11].

In this context, this chapter aims to discuss practical aspects for implementing a functional quality management system at materials science laboratories. Considering this approach, in the reminder of this section a brief discussion on the basic principles of ISO/IEC 17025 will be presented, followed by a practical approach related to the main aspects of quality management system implementation.

The aspects discussed in the aforementioned method are mainly associated with the experience of the authors regarding the implementation of an ISO/IEC 17025 oriented quality manage‐ ment system at the Materials Metrology Division (Dimat) of the National Institute of Metrol‐ ogy, Quality and Technology (Inmetro), the Brazilian National Metrology Institute (NMI).

#### **1.1. ISO/IEC 17025—General requirements for the competence of testing and calibration laboratories standard: basic principles**

ISO/IEC 17025 specifies the basic requirements for the competence verification of laboratories carrying out testing and calibration activities, focused in meeting customer expectations and keeping organized laboratory records and documents. These requirements relate to most, if not to all, the laboratory activities concerning testing and calibration services provided by the laboratory, from the control of documents and records to the technical procedures standardi‐ zation. The standard separates these requirements into two wide classes: management requirements and technical requirements. The former will be briefly described in this section, since complete information can be found in the ISO/IEC 17025.

Particularly interesting cases are the materials science laboratories [7]. Many of these labora‐ tories, originally headed almost exclusively to R&D activities, became interested in providing accredited tests for materials characterization. As a consequence, they should adopt a quality management system based on the ISO/IEC 17025 requirements standard [8]. On the other hand, materials characterization laboratories specialized in providing accredited high‐technology tests are constantly pushed to carry out R&D, in order to improve or develop new processes and services. One way or another, both cases describe scenarios where quality management system must run in an environment where R&D and accredited testing services co‐exist.

Considering this context, implementing quality management systems at materials science laboratories is a challenging task due to several reasons. The first point is the fact that whilst the implementation of an ISO/IEC 17025 oriented quality management system at testing laboratories is a consensus, the advantages and disadvantages of applying quality manage‐ ment concepts to R&D activities are still issues widely discussed by the quality management community. Indeed, the main aspects of this controversy are highlighted in the articles of Mathur‐De‐Vré [9] and Krapp [10]. They state that adding quality management concepts, like metrological traceability, trackability and rational human resources management, must not

In spite of this contradiction, materials science laboratories, but also R&D laboratories of several scientific areas, start to see the standardization promoted by the implementation of a quality management system based on ISO/IEC 17025 standard as a way to reduce costs, reduce tests execution deadlines, satisfy customers and especially add quality to their services and products. More recently, an article published in *Nature* has drawn attention to how quality management and quality assurance concepts could improve R&D activities and results [11]. In this context, this chapter aims to discuss practical aspects for implementing a functional quality management system at materials science laboratories. Considering this approach, in the reminder of this section a brief discussion on the basic principles of ISO/IEC 17025 will be presented, followed by a practical approach related to the main aspects of quality management

The aspects discussed in the aforementioned method are mainly associated with the experience of the authors regarding the implementation of an ISO/IEC 17025 oriented quality manage‐ ment system at the Materials Metrology Division (Dimat) of the National Institute of Metrol‐ ogy, Quality and Technology (Inmetro), the Brazilian National Metrology Institute (NMI).

**1.1. ISO/IEC 17025—General requirements for the competence of testing and calibration**

ISO/IEC 17025 specifies the basic requirements for the competence verification of laboratories carrying out testing and calibration activities, focused in meeting customer expectations and keeping organized laboratory records and documents. These requirements relate to most, if not to all, the laboratory activities concerning testing and calibration services provided by the laboratory, from the control of documents and records to the technical procedures standardi‐ zation. The standard separates these requirements into two wide classes: management

restrain the flexibility necessary in order to carry out R&D activities.

24 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

system implementation.

**laboratories standard: basic principles**

Management requirements are mainly related to organization, control and update of docu‐ ments, analysis of contracts and monitoring optimization of the quality management system. Its main topics are summarized as follows:


On the other hand, technical requirements are directly related to testing and calibration procedures, as summarized below:


Up to this date, ISO/IEC 17025 standard was last revised and confirmed by ISO in 2010 and is currently under a new revision process [12]. Accordingly to some stakeholders, several aspects should be reviewed in order to modernize the standard concepts and specifications, as, among others [13] quality system management designing based on performance and process, trackability, modernization of requirements related to software and electronic records and the evaluation of other standards (as ISO/IEC 17065—conformity assessment—requirements for bodies certifying products, processes and services, for example) relevance to ISO/IEC 17025.

Although these requirements are separated into two main classes, they should not be seen as isolated. As a matter of fact, it is easier to see the relation between them if one considers the basic system development concept underneath the implementation of a quality management system based on a requirement standard. This concept is represented in **Figure 1**.

Any quality management system such as the ISO/IEC 17025 is based on a tripod formed by policies, procedures and records.

Policies are the platform for the development of the system. They must clearly dispose the figures responsible for the different processes that constitute the management system itself, as well as the basic principles that encompass these processes. As an example, the policy for documents control may dispose that update and access control is the responsibility of a centralized system (a document control centre) for this very purpose, whilst the production of technical documents is the responsibility of the laboratory staff. Based on this simple policy, documents control centre and laboratories can, together, develop procedures to ensure the correct access, update and utilization of documents.

As per ISO‐specific nomenclature, all policies must be documented. On the other hand, procedures describe how a quality management process must be done and how to register its execution. They apply to most of the processes carried out in a laboratory, stretching from technical procedures for testing and calibration to procedures about documents and records production, contracts and suppliers' evaluation, and quality assurance procedures, to name but a few.

Key Aspects for Implementing ISO/IEC 17025 Quality Management Systems at Materials Science Laboratories http://dx.doi.org/10.5772/66100 27

**•** Measurement traceability: deals with traceability via calibration or use of certified reference

**•** Sampling: involves the use of standard sampling methods adequate to different sort of

**•** Handling of testing and calibration items: covers definition of procedures to identify and

**•** Results quality assurance: covers a programme of actions related to the quality assurance of obtained results. Such a programme may contain actions, for example, participation in

**•** Reporting results: this topic consists of the procedures required for communicating tests

Up to this date, ISO/IEC 17025 standard was last revised and confirmed by ISO in 2010 and is currently under a new revision process [12]. Accordingly to some stakeholders, several aspects should be reviewed in order to modernize the standard concepts and specifications, as, among others [13] quality system management designing based on performance and process, trackability, modernization of requirements related to software and electronic records and the evaluation of other standards (as ISO/IEC 17065—conformity assessment—requirements for bodies certifying products, processes and services, for example) relevance to ISO/IEC 17025.

Although these requirements are separated into two main classes, they should not be seen as isolated. As a matter of fact, it is easier to see the relation between them if one considers the basic system development concept underneath the implementation of a quality management

Any quality management system such as the ISO/IEC 17025 is based on a tripod formed by

Policies are the platform for the development of the system. They must clearly dispose the figures responsible for the different processes that constitute the management system itself, as well as the basic principles that encompass these processes. As an example, the policy for documents control may dispose that update and access control is the responsibility of a centralized system (a document control centre) for this very purpose, whilst the production of technical documents is the responsibility of the laboratory staff. Based on this simple policy, documents control centre and laboratories can, together, develop procedures to ensure the

As per ISO‐specific nomenclature, all policies must be documented. On the other hand, procedures describe how a quality management process must be done and how to register its execution. They apply to most of the processes carried out in a laboratory, stretching from technical procedures for testing and calibration to procedures about documents and records production, contracts and suppliers' evaluation, and quality assurance procedures, to name

system based on a requirement standard. This concept is represented in **Figure 1**.

handle samples to tests, as well as records of trackability;

26 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

inter‐laboratory comparison or proficiency‐testing programmes; and

materials or standards;

and calibration results to the clients.

policies, procedures and records.

but a few.

correct access, update and utilization of documents.

materials;

**Figure 1.** Basic concepts underneath a quality management system based on a requirements standard.

Finally, it is worth to mention that the records comprise not only testing and calibration results, but also complete status of laboratory equipment, staff training and ambient conditions. In other words, the records consist of all activities related to the quality management system. The way the recording is performed, stored and made available is usually specified in procedures. Thus, all requirements should be considered in unison so as to design a functional quality management system.

One last, but fundamental aspect related to the policies‐procedures‐records tripod is that although it constitutes the base (policies) and the tools (procedures and records) for the quality management system operation, the system itself is under constant improvement. In this way, procedures and even policies are constantly evolving in order to simply correct eventual non‐ conforming or to optimize the system. In **Figure 1**, this characteristic is depicted by the central cycle representing the 'plan‐do‐act‐check' system (PDCA).

All the requirements disposed in ISO/IEC 17025 must be fulfilled when implementing the quality management system. However, the extensive discussion of each one of the require‐ ments is not the intent of this chapter. Instead, this chapter will cover the aspects regarded as key requirements for implementing a functional and flexible quality management system at materials science laboratories. Section 3 addresses some practical topics concerning strategies for implementing a quality management system and, lastly, final considerations are presented in Section 4.

## **2. Key matters in quality management systems in materials science laboratories**

#### **2.1. Organization of laboratories: the quality management system set out**

Implementation of a quality management system at materials science laboratories involves a clearly specified chronogram. Additionally, responsibilities of the laboratory staff members should be defined in detail in order to organize the process.

However, a critical question concerning the necessary units for work organisation whilst implementing a functional and lissom quality management system must be answered.

Most of materials science laboratories make use of several experimental techniques to charac‐ terize different materials properties, such as structure, chemical composition, and thermo‐ physical and mechanical characteristics among others. Therefore, as the scope of techniques associated with materials science covers a wide area, it is not rare that the same organization (company or laboratory) holds multiple analysis techniques that may differ from applications, scale or sort of materials. In addition, each one of them demands very specific requirements concerning facilities and staff.

One possible approach is to gather similar experimental techniques in the specific laboratory units. The quality management team, in co‐operation with the technical staff, may choose similar and/or complementary techniques to compose each unit. Another approach is to organize the laboratories driven by external demand. For example, an institute/company for testing building materials may organize multiple laboratories following the regulatory compliance, such as energy efficiency, thermal comfort and fire safety. Another common case is that of laboratories inside companies producing materials, designed for materials quality control. In such cases, the laboratories can be organized according to specific production needs.

On the other hand, not only external demands, but also internal factors should be considered. The staff size must be compatible with the experimental complexities and tests demands. Besides, physical and financial aspects are also important, because a laboratory divided into multiple locations without financial autonomy may result in an unnecessarily complicated management system that could hamper the management.

Furthermore, if the laboratory belongs to a company or may form another legal entity, the organizational structure must ensure that duties are clearly assigned, eliminating the possi‐ bility of conflict of interests between the laboratory and any other operating units.

Considering these aspects, it is the authors' view that a rational organization of technical infrastructure and equipment in a minimal, but necessary, number of different laboratories or units can minimize financial and human resources costs. If the structure is spread among many different laboratories, the production of documents may turn into a burden, resulting in a highly complex and oversized quality management system, presenting, for instance, redun‐ dant procedural documents in different laboratories. Otherwise, the distribution of the structure between few laboratories may produce a frame where many different techniques are condensed in a single unit, creating very complex laboratories which will be hard to manage. Besides the practical aspects presented above, it is equally important to understand how the customer sees the company's activities. As an example, consider a client who wants tests for compliance to some specific standard or regulation, as is the case of characterization of thermal conductivity for insulator materials. In this case, it will be much easier to maintain a quality system and to organize records and documents when laboratories are directly related to the sets of tests performed for each regulation. However, if the customers see the organization as a centre of excellence in materials science, organizing laboratories by similar techniques could be more appropriate.

As one can see, the approach adopted to organize the technical infrastructure in laboratories or units is a 'tailor made' task that impacts several aspects of the strategy used for implement‐ ing the quality management system. It determines the background for the definition of policies and processes common to all laboratories, such as documents control, services and supplies acquisition, customer service, internal audit, control of records and management of non‐ conforming works and events.

After that technical requirements specific to each laboratory/unit could be easily implemented by combining particularities of each one of them with the common policies and processes, in order to optimize system standardization. In this aspect, a well‐designed organization of laboratories/units automatically shows that keeping some processes common to all laborato‐ ries minimizes the efforts from technical staff in document production and also improves the collaborative process for quality management.

#### **2.2. Control of documents and records**

**2. Key matters in quality management systems in materials science**

Implementation of a quality management system at materials science laboratories involves a clearly specified chronogram. Additionally, responsibilities of the laboratory staff members

However, a critical question concerning the necessary units for work organisation whilst implementing a functional and lissom quality management system must be answered.

Most of materials science laboratories make use of several experimental techniques to charac‐ terize different materials properties, such as structure, chemical composition, and thermo‐ physical and mechanical characteristics among others. Therefore, as the scope of techniques associated with materials science covers a wide area, it is not rare that the same organization (company or laboratory) holds multiple analysis techniques that may differ from applications, scale or sort of materials. In addition, each one of them demands very specific requirements

One possible approach is to gather similar experimental techniques in the specific laboratory units. The quality management team, in co‐operation with the technical staff, may choose similar and/or complementary techniques to compose each unit. Another approach is to organize the laboratories driven by external demand. For example, an institute/company for testing building materials may organize multiple laboratories following the regulatory compliance, such as energy efficiency, thermal comfort and fire safety. Another common case is that of laboratories inside companies producing materials, designed for materials quality control. In such cases, the laboratories can be organized according to specific production needs. On the other hand, not only external demands, but also internal factors should be considered. The staff size must be compatible with the experimental complexities and tests demands. Besides, physical and financial aspects are also important, because a laboratory divided into multiple locations without financial autonomy may result in an unnecessarily complicated

Furthermore, if the laboratory belongs to a company or may form another legal entity, the organizational structure must ensure that duties are clearly assigned, eliminating the possi‐

Considering these aspects, it is the authors' view that a rational organization of technical infrastructure and equipment in a minimal, but necessary, number of different laboratories or units can minimize financial and human resources costs. If the structure is spread among many different laboratories, the production of documents may turn into a burden, resulting in a highly complex and oversized quality management system, presenting, for instance, redun‐ dant procedural documents in different laboratories. Otherwise, the distribution of the structure between few laboratories may produce a frame where many different techniques are condensed in a single unit, creating very complex laboratories which will be hard to manage.

bility of conflict of interests between the laboratory and any other operating units.

**2.1. Organization of laboratories: the quality management system set out**

should be defined in detail in order to organize the process.

28 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

management system that could hamper the management.

**laboratories**

concerning facilities and staff.

The purpose of a document control system is to deal with the large amount of documents, such as procedures, reports and forms generated by the quality management system. In such a system, any given document must be considered as a unique element. It must have a unique identifier, like a name or code, and each one of these documents has a lifetime in the system, from its release to its cancel dates. The documents *per se* evolve over time, necessitating thus, a revision. Therefore, a functional documents control system must keep track documenting the corresponding characteristics.

The system must control not only the internal procedures developed within the company or laboratory, but it must also include instrument manuals, software, drawings, standards and regulations as well as external source documents (i.e. national or international standards for testing of materials). This last class of documents usually demands special attention, since the revision of such standards by the responsible organizations must be periodically monitored by the laboratory, in order to avoid the use of obsolete external standards.

In practice, the complexity of such a system depends on each case. For small laboratories carrying out a small number of tests, an electronic spread sheet could be enough for document management. However, for larger corporations with several laboratories, a system running a consolidated documents database allowing relational searches could be a proper choice. Apart from its complexity, the system must keep track of a certain number of document characteristics such as:


Additionally, invalid and obsolete versions of documents must be clearly identified in the system, in order to avoid misuse. Finally, the system must provide access to all its documen‐ tation to the users.

Document management goes beyond an effective document control system. The standardiza‐ tion and documentation of processes is a tool to make activities much more effective and efficient. Documents should be easy to read and simple to use. They must disseminate knowledge within the organization. They could be hierarchical and should be referred to each other in such a way that it improves the document writing process by reducing the amount of rework. They should be based on consensus of those involved in the process. Each document must describe the process accurately, making use of figures, tables and flowcharts as and where necessary. They must be written with its practical use in mind and focused on being easily understood.

Records must be also included in a management system. Several records must be kept at the laboratory level—such as testing results—and must be organized considering the particular‐ ities of each laboratory or unit.

There are other records that can be kept organized outside the laboratory to provide easy access, as follows:


The record management system may also keep track of measures of productivity, assisting management decisions. It is the authors' view that control of records may impose bottlenecks when a quality management system is applied to R&D activities. Indeed, during R&D projects execution, detailed investigation of a considerable number of parameters is carried out. These activities result in a large number of preliminary results that will not be included in the research outcome. However, even these preliminary results must be accounted as controlled records by the quality management system.

All these aspects make designing a system to control documents and records a customized task, which must take into account specific characteristics of the company/laboratory, as the mechanism to provide rapid access to documents and eventual access privileges. However, once a system of documents and records management is working properly, it makes labora‐ tory‐daily activities and follow‐up tasks easier, since the information is always organized and readily available.

#### **2.3. Customer relationship service**

**•** release date;

tation to the users.

understood.

access, as follows: **•** meeting minutes,

ities of each laboratory or unit.

**•** management review minutes,

**•** personnel documentation and

the quality management system.

**•** test item handling records.

**•** implementing reports,

**•** item registration,

**•** revision date (from the first to the later revision);

30 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**•** unique identification for each revision (revision number); and

**•** copies of all revisions, authors, reviewers, and the person(s) who approved each document. Additionally, invalid and obsolete versions of documents must be clearly identified in the system, in order to avoid misuse. Finally, the system must provide access to all its documen‐

Document management goes beyond an effective document control system. The standardiza‐ tion and documentation of processes is a tool to make activities much more effective and efficient. Documents should be easy to read and simple to use. They must disseminate knowledge within the organization. They could be hierarchical and should be referred to each other in such a way that it improves the document writing process by reducing the amount of rework. They should be based on consensus of those involved in the process. Each document must describe the process accurately, making use of figures, tables and flowcharts as and where necessary. They must be written with its practical use in mind and focused on being easily

Records must be also included in a management system. Several records must be kept at the laboratory level—such as testing results—and must be organized considering the particular‐

There are other records that can be kept organized outside the laboratory to provide easy

The record management system may also keep track of measures of productivity, assisting management decisions. It is the authors' view that control of records may impose bottlenecks when a quality management system is applied to R&D activities. Indeed, during R&D projects execution, detailed investigation of a considerable number of parameters is carried out. These activities result in a large number of preliminary results that will not be included in the research outcome. However, even these preliminary results must be accounted as controlled records by

All these aspects make designing a system to control documents and records a customized task, which must take into account specific characteristics of the company/laboratory, as the Customer service is a common requirement for all testing laboratories and the usual approach perfectly applies to materials science laboratories. When a company or institution structure consists of several laboratories, the usual approach can be applied. Establishing a specialized sector for customer service is a suitable policy that allows laboratories technical staff to spend more time for laboratories activities (testing services and R&D activities). Additionally, by minimizing the direct contact between the client and the technical staff, the customer service sector may also avoid or at least reduce some unpleasant situations, like external pressure for anticipating deadlines or questions about price. The customer service sector should be assigned to handle all the communication between the company and the clients, from the first commer‐ cial contact to the finalization of the contracted service. It is not uncommon that the same channel used for customer service also fulfils the function of customer's claims channel. Although these observations are pretty obvious, it is addressed here because it is crucial for R&D laboratories wishing to start providing accredited tests to properly take care of costumer services from the start.

Although customer service sector limits the direct contact between the client and the technical staff, in the case of materials science laboratories this direct contact should be done in order to clarify technical aspects related to the test as demanded by the costumer. These technical aspects normally relate to the use of specific experimental procedures and an eventual deviation from standard testing procedures in order to suit customer needs. It may also be due to some specific characteristic of the tested material. It is important to mention that when the laboratory uses methods other than standardized ones, even due to customer's request, such methods must be validated by the laboratory technical staff.

#### **2.4. Calibration, traceability to the SI and quality assurance of test results**

The most relevant impact of implementing a quality management system in any laboratory developing tests and R&D is the quality assurance of the experimental results. This assurance is important not only to a client interested in materials properties characterization, but also to the R&D activities, since traceability and reproducibility of results are deemed as key factors in science and technology. In this aspect, the implementation of a quality management system based on the ISO/IEC 17205 shall demonstrate confidence in experimental results. From a technical standpoint, the establishment of calibration and traceability to appropriate meas‐ urement standards is fundamental, and consequently the quality assurance can be demon‐ strated. As such, this section discusses how calibration and traceability can be handled in materials science laboratories in order to improve quality assurance of experimental results.

ISO/IEC 17025 standard requires that all equipment used for testing or calibration must be calibrated in order to assure the quality of results. This applies also to subsidiary measure‐ ments, such as environmental conditions monitoring, as long as they have a significant contribution to the total uncertainty of the results. An useful example is the International Vocabulary of Metrology (VIM) [14], at page 28, defines calibration as '… operation that, under specified conditions, establishes a relation between the quantity values with measurement uncertainties provided by measurement standards and corresponding indications with associated measurement uncertainties and, in a second step, uses this information to establish a relation for obtaining a measurement result from an indication'. The purpose of the calibra‐ tion is not simply to ensure that the equipment is providing a correct result, but most of all, to guarantee a link of this measurement with the unit definitions in the international system of units (SI). This link, that establishes a relation between the result of a measurement and the abstract SI units, is called metrological traceability.

This traceability, however, can also refer to a measurement procedure including the measure‐ ment unit or a measurement standard. The concept of traceability is implied when dealing with traditional calibration procedures, such as the calibration of a scale. In this case, a standard weight is used to calibrate the scale in a simple procedure. Therefore, traceability is secured by an unbroken chain of calibrations tracing back to the international mass standard, which is unique and is the physical realization of the kilogram unit. This simple procedure, however, is far from the reality of an engineering materials testing laboratory.

When dealing with tests of materials, some of their properties may not be directly linked to the SI through a simple metrological traceability chain [15]. Typically, even a simple measure‐ ment of a material property involves many different measurements and links to several SI units to guarantee the metrological traceability. An useful example is the thermal conductivity, whose unit is [W/m K] or equivalently [m kg s3 K], measured by the GHP method. In this case, the metrological traceability to four different units needs to be provided, which in turn, requires calibration of several instruments used in the process. Although this is the primary method to establish the traceability to SI, this is a very laborious task and, in some cases, it is hard to be used for some materials characterization techniques. In these cases, alternative approaches are used [16].

Another topic that makes testing of materials not so easy task is that some properties of a material are not intrinsic but dependent on the measurement procedure, which is called a procedural property. This means that the measurement of a given material property can return different values depending on the measurement method. A typical example is the hardness. In this case, the traceability is guaranteed by not only the calibration of the instruments but also by a documented standard defining the method. Finally, sometimes a measurement cannot be linked to SI, as a result of the measurement is not an SI unit but a classification on an appropriate numerical scale. Once again, the metrological traceability in this case is given by a measurement standard procedure and some reference materials. An example is the Mohs Hardness scale.

In all cases, the ISO/IEC 17025 standard gives alternatives to foster confidence in measurements by establishing the traceability to appropriate measurement standards. Those alternatives are as follows:


ISO/IEC 17025 standard requires that all equipment used for testing or calibration must be calibrated in order to assure the quality of results. This applies also to subsidiary measure‐ ments, such as environmental conditions monitoring, as long as they have a significant contribution to the total uncertainty of the results. An useful example is the International Vocabulary of Metrology (VIM) [14], at page 28, defines calibration as '… operation that, under specified conditions, establishes a relation between the quantity values with measurement uncertainties provided by measurement standards and corresponding indications with associated measurement uncertainties and, in a second step, uses this information to establish a relation for obtaining a measurement result from an indication'. The purpose of the calibra‐ tion is not simply to ensure that the equipment is providing a correct result, but most of all, to guarantee a link of this measurement with the unit definitions in the international system of units (SI). This link, that establishes a relation between the result of a measurement and the

This traceability, however, can also refer to a measurement procedure including the measure‐ ment unit or a measurement standard. The concept of traceability is implied when dealing with traditional calibration procedures, such as the calibration of a scale. In this case, a standard weight is used to calibrate the scale in a simple procedure. Therefore, traceability is secured by an unbroken chain of calibrations tracing back to the international mass standard, which is unique and is the physical realization of the kilogram unit. This simple procedure, however,

When dealing with tests of materials, some of their properties may not be directly linked to the SI through a simple metrological traceability chain [15]. Typically, even a simple measure‐ ment of a material property involves many different measurements and links to several SI units to guarantee the metrological traceability. An useful example is the thermal conductivity,

the metrological traceability to four different units needs to be provided, which in turn, requires calibration of several instruments used in the process. Although this is the primary method to establish the traceability to SI, this is a very laborious task and, in some cases, it is hard to be used for some materials characterization techniques. In these cases, alternative

Another topic that makes testing of materials not so easy task is that some properties of a material are not intrinsic but dependent on the measurement procedure, which is called a procedural property. This means that the measurement of a given material property can return different values depending on the measurement method. A typical example is the hardness. In this case, the traceability is guaranteed by not only the calibration of the instruments but also by a documented standard defining the method. Finally, sometimes a measurement cannot be linked to SI, as a result of the measurement is not an SI unit but a classification on an appropriate numerical scale. Once again, the metrological traceability in this case is given by a measurement standard procedure and some reference materials. An example is the Mohs

K], measured by the GHP method. In this case,

abstract SI units, is called metrological traceability.

32 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

whose unit is [W/m K] or equivalently [m kg s3

approaches are used [16].

Hardness scale.

is far from the reality of an engineering materials testing laboratory.

**•** participation in suitable inter‐laboratory comparisons where possible.

For materials testing, the use of certified reference materials (CRM) is typically the preferred choice. It is important that the reference material should be accompanied by a document, issued by a recognized organization. The latter provides values of specific properties with associated uncertainty and metrological traceability, using validated procedures. It is worth mentioning at this point that, when dealing with a specific material, there are many of its properties that can be assessed and used as a reference in a measurement, but the CRM is typically certified for just one of its properties of interest.

The certification of this property can be achieved through a measurement using a primary method, with the lowest uncertainty and with the necessary traceability. This reference material is then used to provide the traceability to the measurements of this property in a much simpler way. The drawback is that the measurement uncertainty is a bit higher than the one obtained by primary methods, but most of the time this is low enough for the researcher's purpose.

There are still cases where one cannot certify a reference material using a primary method. In this case, the certification is achieved by consensus. This means that different laboratories make the measurement using a given protocol, and the mean value is used as a reference value for that given property.

A laboratory participating in such a comparison can (or cannot) demonstrate their competence and measurement capability, without having to provide traceability to SI for their results. This is similar to laboratories that realize the SI units, and therefore, have no physical standard that can be used to calibrate their instrument. Those laboratories undergo international inter‐ laboratory comparisons with other laboratories in the same status, in order to mutually guarantee their measurement capabilities. In the specific case of national metrology institutes, there is the possibility of participation in key comparisons organized by the Bureau Interna‐ tional de Poids et Mesures (BIPM) [17].

If there is no CRM available or, if no inter‐laboratory comparisons have been performed, an alternative way to assure the quality of test and calibration is to periodically perform intra‐ laboratory comparisons. In an intra‐laboratory test, two or more researchers perform the material measurement, using—preferentially—a CRM and calibrated equipment, and the results are compared. With this practice, it is possible to analyse data and detect trends that can influence the result and uncertainty.

At BIPM there are several consultative committees (CC) that are responsible for the base SI units, namely:


Indeed, the same applies for the derived units or for specific fields such as acoustics, ultrasound and ionizing radiation. There is, so far, no specific CC for materials. BIPM understands that any material property can be treated under one of the current CCs. For example, thermal conductivity is treated under the consultative committee for thermometry (CCT).

An important forum for discussion on measurement methods and procedures are the ISO technical committees (TC). Once again there is no specific TC for materials metrology, but there are so many TCs; it is unlikely that one given property of a material is not treated in one of the committees. ISO is a normative organization and as such it is concerned with publishing standards, either for a material testing or for method standardization. Typically, a normative standard is produced when there is already a well‐established consensus in the validity of such a method or test that one wants to standardize. However, one can propose inside a TC a study group to develop and validate a new method before writing the standard, taking advantage of the internationality and wide extent of the members.

This procedure is however not preferable, since it may take too long to fully develop new methods, and the ISO time is short and well defined.

A solution to this problem is the pre‐normative forums. One of the most important forums is the Versailles project on advanced materials and standards (VAMAS), whose main objective is to '…promote world trade by innovation and adoption of advanced materials through international collaborations that provide the technical basis for harmonization of measurement methods, leading to best practices and standards' [18]. The VAMAS has been founded under the auspices of the G7 economic summit in 1982 and includes a number of representatives from many countries in the world. The VAMAS consists of the technical working areas (TWA) dealing with a class of materials or a given property of materials, such as polymer composites, mechanical properties of thin films and nanoparticle populations. In every TWA, there are many projects that deal with specific subjects. In this way, one can propose new measurement methods or new testing for new materials, run inter‐laboratory comparisons with the collab‐ oration of different national metrology institutes and also other stakeholders from industry and academia.

The VAMAS is also liaising with the BIPM in the development of materials metrology. While VAMAS can propose new procedures and measurement methods, BIPM would run the inter‐ laboratory comparisons among the NMIs using their well‐established system of key compar‐ isons. Finally, if the project is successful, one might forward it to the ISO and develop a standard out of it, since it has already been tested by a worldwide community.

In order to manage equipment and reference materials, calibration routine and records, it is critical to have a calibration programme. In such a programme, the laboratory shall control the calibration status of the equipment and the reference materials that are relevant for the testing procedure. Different equipment presents different needs regarding its calibration periodicity. For instance, a scale or ruler can be calibrated on an annual basis, because their daily use does not change their calibration condition, if well conserved and appropriately handled. On the other side, some types of apparatus may require a calibration before the beginning of each measurement, as their calibration status may not be valid for a long period. Finally, for multi‐ faceted equipment used in typical material testing laboratories, their calibration is too complex to be done on a periodical basis. One must, however, confirm that performance properties or legal requirements of the measuring system are achieved. This is called verification. When possible, both calibration and verification of the equipment should be done with the use of reference standards or certified reference materials. It is also quite important to verify the status of any apparatus when it goes through maintenance or when it is used outside the laboratory's environment.

**•** length, **•** mass, **•** time,

**•** electric current,

and academia.

**•** thermodynamic temperature,

**•** amount of substance and luminous intensity.

34 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

of the internationality and wide extent of the members.

methods, and the ISO time is short and well defined.

Indeed, the same applies for the derived units or for specific fields such as acoustics, ultrasound and ionizing radiation. There is, so far, no specific CC for materials. BIPM understands that any material property can be treated under one of the current CCs. For example, thermal

An important forum for discussion on measurement methods and procedures are the ISO technical committees (TC). Once again there is no specific TC for materials metrology, but there are so many TCs; it is unlikely that one given property of a material is not treated in one of the committees. ISO is a normative organization and as such it is concerned with publishing standards, either for a material testing or for method standardization. Typically, a normative standard is produced when there is already a well‐established consensus in the validity of such a method or test that one wants to standardize. However, one can propose inside a TC a study group to develop and validate a new method before writing the standard, taking advantage

This procedure is however not preferable, since it may take too long to fully develop new

A solution to this problem is the pre‐normative forums. One of the most important forums is the Versailles project on advanced materials and standards (VAMAS), whose main objective is to '…promote world trade by innovation and adoption of advanced materials through international collaborations that provide the technical basis for harmonization of measurement methods, leading to best practices and standards' [18]. The VAMAS has been founded under the auspices of the G7 economic summit in 1982 and includes a number of representatives from many countries in the world. The VAMAS consists of the technical working areas (TWA) dealing with a class of materials or a given property of materials, such as polymer composites, mechanical properties of thin films and nanoparticle populations. In every TWA, there are many projects that deal with specific subjects. In this way, one can propose new measurement methods or new testing for new materials, run inter‐laboratory comparisons with the collab‐ oration of different national metrology institutes and also other stakeholders from industry

The VAMAS is also liaising with the BIPM in the development of materials metrology. While VAMAS can propose new procedures and measurement methods, BIPM would run the inter‐ laboratory comparisons among the NMIs using their well‐established system of key compar‐ isons. Finally, if the project is successful, one might forward it to the ISO and develop a standard

out of it, since it has already been tested by a worldwide community.

conductivity is treated under the consultative committee for thermometry (CCT).

Not only equipment, but reference standards and CRM must also have a systematic control of their calibration situation or validity of its certification. While reference standards can be re‐ calibrated on a periodic basis just like any equipment, CRMs normally have a short life span that guarantees their properties. After its expiration, the laboratory must then acquire new CRMs or, when possible, get its re‐certification. Particularly to some materials, due to their stable properties, the validity of their certification can be indefinite. As an example hereto, the alumina powder (corundum) reference material for quantitative phase analysis using the powder diffraction method is mentioned. This reference material has been developed by the NIST [19] whose certification is valid indefinitely as long as it is stored and handled according to its certificate.

Additionally, in some particular cases, it is not possible to calibrate the instrument and there is no CRM available for instrument verification. In such cases, ISO/IEC 17025 recommends to repeat the tests using samples already tested and retained in the laboratory and/or to compare the results of tests performed using different methods, in order to provide some evidence of quality assurance.

The routinely monitoring of calibrations or verifications can be recorded with the use of control charts that provide an easy way to examine the overall status of the equipment or reference standards and identify eventual trends that may affect the results of tests. In order to assure the quality of results, it is important that the laboratory defines and documents the acceptance criteria of the calibrations of equipment, reference standards, and the certificates of CRMs. The control charts are viewed as a very valuable tool for this task. It is worth mentioning at this point that activities related to quality assurance not only involve technical aspects, but also organizational elements that permeate work at all stages. Thus, a quality system with good documentation, internal and external auditing and records (equipment maintenance, correc‐ tive/preventive actions, etc.) strongly supports quality assurance.

#### **2.5. Evaluation of uncertainty of measurements**

Measurement uncertainty is a quantitative indication of quality for measurement results. When the uncertainty related to a result is not declared, this result cannot be compared with specified reference values or standards. As a matter of fact, even results from different tests related to the same material/sample are hard to compare without evaluation of uncertainty. Uncertainty evaluation is essential to guarantee the metrological traceability of measurement results and to ensure they are accurate and reliable. Additionally, measurement uncertainty must be accounted for whenever a decision has to be taken based on measurement results.

In this context, progressive globalization of markets pushed for the use of a standard procedure for evaluating uncertainty of measurements, in order to assure comparability of results and, consequently, mutual recognition in metrology. In this aspect, laboratories accredited under the ISO/IEC 17025 standard aiming to demonstrate their technical competence and the ability to properly operate their management systems are required to evaluate uncertainty of their measurement results. The two main approaches commonly used for evaluating measurement uncertainty are the bottom-up and the top-down and both of them will be briefly discussed here.

The bottom-up approach is more often used for classic physical metrology systems, such as mass or dimension measurements. It involves using a model equation (the one used for the calculation of the measurand) as a starting point and then considers all individual uncertainty contributions. This is the approach that is well described in the guide to the expression of uncertainty in measurement (GUM) [20] and it uses the law of propagation of uncertainties as its base. The GUM is a guide to uncertainty evaluation and it is the more widely accepted approach for uncertainty evaluation in metrology studies regarding the comparison of results.

Considering the approach described in GUM, the measurand is described as a function of several variables called input quantities. For each input quantity, the sources of uncertainty should be evaluated, quantified, modelled as a random variable, and then classified as being type A or type B.

Type A uncertainties are the ones evaluated by statistical methods. Sources of uncertainty evaluated using non-statistical methods such as documented values or elicited by expertise are called type B uncertainties. For example, during a simple mass measurement, the repetition and uncertainty from calibration standards are sources of uncertainty. The former is Type A uncertainty and the latter is Type B uncertainty. However, an accurate measurement could compute the buoyancy effect and new sources of uncertainty should be taken into account such as air density changes arising from temperature or pressure variation. It is worth mentioning at this point that the measurand model plays a critical role in uncertainty evaluation.

All sources of uncertainty can be communicated using an Ishikawa diagram [21], also known as the fishbone diagram. The measurand plays the role as a principal bone and each input quantity as an adjacent bone. Each input quantity should have its uncertainty sources attached to it. The fishbone diagram gives in a single chart a qualitative summary of all sources of uncertainty for a given measurement model.

After quantifying the sources of uncertainty, the partial derivative of the measurand model with respect to each input quantity should be calculated. The partial derivative is called sensitivity coefficient. The standard deviation of each uncertainty source multiplied by the sensitivity coefficient gives the standard uncertainty. Considering the result obtained for a given measurand as a function of *f n* independent variables. The uncertainty can be written as:

**2.5. Evaluation of uncertainty of measurements**

36 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

here.

type A or type B.

uncertainty for a given measurement model.

tion.

Measurement uncertainty is a quantitative indication of quality for measurement results. When the uncertainty related to a result is not declared, this result cannot be compared with specified reference values or standards. As a matter of fact, even results from different tests related to the same material/sample are hard to compare without evaluation of uncertainty. Uncertainty evaluation is essential to guarantee the metrological traceability of measurement results and to ensure they are accurate and reliable. Additionally, measurement uncertainty must be accounted for whenever a decision has to be taken based on measurement results. In this context, progressive globalization of markets pushed for the use of a standard procedure for evaluating uncertainty of measurements, in order to assure comparability of results and, consequently, mutual recognition in metrology. In this aspect, laboratories accredited under the ISO/IEC 17025 standard aiming to demonstrate their technical competence and the ability to properly operate their management systems are required to evaluate uncertainty of their measurement results. The two main approaches commonly used for evaluating measurement uncertainty are the bottom-up and the top-down and both of them will be briefly discussed

The bottom-up approach is more often used for classic physical metrology systems, such as mass or dimension measurements. It involves using a model equation (the one used for the calculation of the measurand) as a starting point and then considers all individual uncertainty contributions. This is the approach that is well described in the guide to the expression of uncertainty in measurement (GUM) [20] and it uses the law of propagation of uncertainties as its base. The GUM is a guide to uncertainty evaluation and it is the more widely accepted approach for uncertainty evaluation in metrology studies regarding the comparison of results. Considering the approach described in GUM, the measurand is described as a function of several variables called input quantities. For each input quantity, the sources of uncertainty should be evaluated, quantified, modelled as a random variable, and then classified as being

Type A uncertainties are the ones evaluated by statistical methods. Sources of uncertainty evaluated using non-statistical methods such as documented values or elicited by expertise are called type B uncertainties. For example, during a simple mass measurement, the repetition and uncertainty from calibration standards are sources of uncertainty. The former is Type A uncertainty and the latter is Type B uncertainty. However, an accurate measurement could compute the buoyancy effect and new sources of uncertainty should be taken into account such as air density changes arising from temperature or pressure variation. It is worth mentioning at this point that the measurand model plays a critical role in uncertainty evalua-

All sources of uncertainty can be communicated using an Ishikawa diagram [21], also known as the fishbone diagram. The measurand plays the role as a principal bone and each input quantity as an adjacent bone. Each input quantity should have its uncertainty sources attached to it. The fishbone diagram gives in a single chart a qualitative summary of all sources of

$$\boldsymbol{u}\_{f(\{\boldsymbol{x},\boldsymbol{y}\},\boldsymbol{z},\ldots,\boldsymbol{u})}^{2} = \left(\frac{\partial \boldsymbol{f}}{\partial \mathbf{x}}\right)^{2} \boldsymbol{u}\_{\boldsymbol{x}}^{2} + \left(\frac{\partial \boldsymbol{f}}{\partial \boldsymbol{y}}\right)^{2} \boldsymbol{u}\_{\boldsymbol{y}}^{2} + \left(\frac{\partial \boldsymbol{f}}{\partial \boldsymbol{z}}\right)^{2} \boldsymbol{u}\_{\boldsymbol{z}}^{2} + \ldots, + \left(\frac{\partial \boldsymbol{f}}{\partial \boldsymbol{n}}\right)^{2} \boldsymbol{u}\_{\boldsymbol{n}}^{2} \tag{1}$$

where *uf* is the uncertainty associated with the measurand, the partial derivatives are the sensitivity coefficients and the terms *ui* (*i* = *x*, *y*, *z*, …, *n*) are the uncertainties associated with each one of the independent variables.

The standard uncertainties could be plotted in a bar‐like chart. This kind of chart is known as uncertainty budget and brings to light critical information about the measurement quality. With this chart one can easily assess the major sources of uncertainty that must be reduced in order to enhance the measurement accuracy. This last aspect points out that GUM methodol‐ ogy is also a management tool for continuous improvement.

In materials science measurements, the bottom‐up approach is suitable for limited tests where it is possible to address the measurand via a liable model. Some cases where bottom‐up approaches can be applied are the measurement of specific area by gas adsorption [22] and measurement of thermal conductivity by the guarded hot plate (GHP) method [23].

For cases where bottom‐up approach does not fit or its application is much complex, the top‐ down approach is a suitable tool for estimating uncertainty associated with a measurand. The top‐down method is a phenomenological approach that is based on the validation and the quality control tests results, assuming that these results: (i) cover all the influence factors and (ii) are representative for all measurements. In this approach, which is described in ISO 5725 [24], uncertainty will always have components evaluated from reproducibility and bias.

The top‐down approach is more often used for chemical, biology and materials measurements when some relevant quantities cannot be addressed in a mathematical model. For example, the top‐down method was used by De Temmerman et al. [25, 26] to evaluate the uncertainty associated with the measurement of particle size using transmission electron microscopy (TEM).

To perform the top‐down approach, it is necessary to use an experimental design that addresses the method variability and a set of reference materials to estimate the bias component. This approach could be expensive and time consuming due to the large number of measurements that must be carried out in order to explore the influence of all the relevant experimental parameters on the results. One example is the use of the top‐down approach to evaluate the uncertainty associated with purity analysis by using differential scanning calorimetry (DSC) [27]. Furthermore, in some cases, the applied procedure could become ineffective if future tests must be performed under experimental conditions not contemplated in the experimental design used for uncertainty evaluation.

Besides the issues related to the use of different approaches for estimating uncertainties, measurements of material properties have some particularities that must be taken into account when evaluating their uncertainties. One of them is that these measurements are often made to give representative measurand values related to a considerable amount of material, such as a lot. In addition, in surface analysis there are some cases in which the measurement is made in a local area of a specimen instead of the material as a whole. In these cases, measured values scatter not only by reproducibility of the measurements, but also by possible non‐uniformity of the material [28]. These two components are eventually merged when estimating uncer‐ tainty for a reference material. Additionally, it is worth mentioning at this point that (Section 2.4) material properties can be classified in two categories: intrinsic properties and procedural properties.

Intrinsic properties are inherent to the material and its value does not depend on the meas‐ urement procedure. On the other hand, procedural properties are totally dependent on the measurement procedure. So, in this last case, two or more property values can only be compared if the measurement procedures have been exactly the same.

## **3. Practical view for implementing quality management systems**

#### **3.1. Strategies for implementation and its progress monitoring**

There are several ways to approach the implementation of a quality management system based on ISO/IEC 17025 in a laboratory. It is the authors' view that most of the time such a system will be implemented in a laboratory that is already in operation. Therefore, it is important that the strategy adopted for implementing the system minimizes the impact of the implementation process on the testing and R&D activities carried out by the laboratory. Furthermore, the strategy shall be flexible and prioritize constant follow‐up in order to avoid repetition of work and optimize the schedule whilst minimizing the cost of the process. In this aspect, there are some points that must always be considered in the adopted strategy.

Once the organizational structure of laboratories/units has been set, as discussed in Section 2.1, the next step is to build a time chart highlighting the requirements of ISO/IEC 17025 standard which will also account for any existing internal requirements specifically applied to the provided testing services. Additionally, this time frame should specify the responsibilities of the laboratory staff members in order to organize the process. For improving the results, production and registration of technical documents related to quality assurance of testing results should be considered as the critical steps of the process. Such documents are technical standards for equipment operation, calibration and maintenance, standards for testing execution and forms used to record testing results and also any procedure related to quality assurance. Indeed, this type of implementation project management was discussed by Silva et al. [7] and, in brief, should be designed considering the following points:

**•** Clear scope definition: scope definition influences all the chronogram steps. It is crucial to clearly select the testing services that will be covered by the quality management system, and, if the laboratory is part of a large company or institute, to identify the quality man‐ agement documents that dispose about laboratory activities. This item may be considered pretty basic and obvious, but these observations are essential to speed up the process;

must be performed under experimental conditions not contemplated in the experimental

Besides the issues related to the use of different approaches for estimating uncertainties, measurements of material properties have some particularities that must be taken into account when evaluating their uncertainties. One of them is that these measurements are often made to give representative measurand values related to a considerable amount of material, such as a lot. In addition, in surface analysis there are some cases in which the measurement is made in a local area of a specimen instead of the material as a whole. In these cases, measured values scatter not only by reproducibility of the measurements, but also by possible non‐uniformity of the material [28]. These two components are eventually merged when estimating uncer‐ tainty for a reference material. Additionally, it is worth mentioning at this point that (Section 2.4) material properties can be classified in two categories: intrinsic properties and procedural

Intrinsic properties are inherent to the material and its value does not depend on the meas‐ urement procedure. On the other hand, procedural properties are totally dependent on the measurement procedure. So, in this last case, two or more property values can only be

There are several ways to approach the implementation of a quality management system based on ISO/IEC 17025 in a laboratory. It is the authors' view that most of the time such a system will be implemented in a laboratory that is already in operation. Therefore, it is important that the strategy adopted for implementing the system minimizes the impact of the implementation process on the testing and R&D activities carried out by the laboratory. Furthermore, the strategy shall be flexible and prioritize constant follow‐up in order to avoid repetition of work and optimize the schedule whilst minimizing the cost of the process. In this aspect, there are

Once the organizational structure of laboratories/units has been set, as discussed in Section 2.1, the next step is to build a time chart highlighting the requirements of ISO/IEC 17025 standard which will also account for any existing internal requirements specifically applied to the provided testing services. Additionally, this time frame should specify the responsibilities of the laboratory staff members in order to organize the process. For improving the results, production and registration of technical documents related to quality assurance of testing results should be considered as the critical steps of the process. Such documents are technical standards for equipment operation, calibration and maintenance, standards for testing execution and forms used to record testing results and also any procedure related to quality assurance. Indeed, this type of implementation project management was discussed by Silva et

compared if the measurement procedures have been exactly the same.

**3.1. Strategies for implementation and its progress monitoring**

some points that must always be considered in the adopted strategy.

al. [7] and, in brief, should be designed considering the following points:

**3. Practical view for implementing quality management systems**

design used for uncertainty evaluation.

38 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

properties.


From those points, the definition of a suitable follow‐up strategy is of paramount importance for optimizing the quality system management implementation process, because it enables 'on‐the‐fly' changes of the adopted strategy and timeframe. When the latter is organized on the basis of requirements and all the tasks are clearly assigned, it is possible to develop a simple and efficient follow‐up programme. It is easier to know how many people are assigned to specific tasks and to verify the relative progress of different tasks. This information is essential to quickly adjust chronograms and even the strategy when necessary.

A functional follow‐up programme is an effective tool for checking the system's maturity determining when it is time to perform a deeper analysis in order to identify fails or gaps in the implementation process. One of the most useful tools to perform this sort of analysis is an internal auditing, which results in an improved diagnostic when compared to the previous follow‐up system. This detailed diagnostic is fundamental to carry out the last adjustments necessary to finish the system implementation. The main steps of the implementation project discussed above and its relationships are shown in **Figure 2**.

**Figure 2.** Main steps of a quality management system implementation project.

A last important practical aspect to be mentioned is the financial impact of implementing a quality management system at a laboratory. A deployed quality management system stand‐ ardizes the processes of the laboratory, demonstrating measurement capability and technical competence. All these aspects add credibility to the results and demonstrate laboratory commitment to the customer. As a consequence, those aspects also translate into financial success by productivity increase, reduced rework and a smaller number of errors. Additionally, standardization of the supplier evaluation process, pricing policy, and careful selection of subcontracts also decreases financial costs. On the other hand, the requirements of a quality system according to ISO/IEC 17025, such as audits, calibrations, and inter‐comparisons bring additional costs to the laboratory. In this aspect, Barradas and Sampaio [29], had shown that investments are feasible and customer satisfaction is improved.

#### **3.2. Implementation time scale**

The time necessary for implementing a quality management system depends basically on the size and complexity of the company/laboratory. It scales with the number of laboratories, testing services and R&D activities that the system will be applied. However, a functional approach to the organization of laboratories and services decreases the time consumed for system's implementation. Indeed, the latter may be regarded as the first step thereto.

For this reason, it is somehow difficult to estimate the time scale necessary for implementing the system without considering a specific case. However, if one considers that a typical structure is composed of a number of different laboratories under a common management centre (a R&D division or department, for example), it is possible to point out some strong time scale bottlenecks:


## **4. Final considerations**

**Figure 2.** Main steps of a quality management system implementation project.

40 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

investments are feasible and customer satisfaction is improved.

**3.2. Implementation time scale**

A last important practical aspect to be mentioned is the financial impact of implementing a quality management system at a laboratory. A deployed quality management system stand‐ ardizes the processes of the laboratory, demonstrating measurement capability and technical competence. All these aspects add credibility to the results and demonstrate laboratory commitment to the customer. As a consequence, those aspects also translate into financial success by productivity increase, reduced rework and a smaller number of errors. Additionally, standardization of the supplier evaluation process, pricing policy, and careful selection of subcontracts also decreases financial costs. On the other hand, the requirements of a quality system according to ISO/IEC 17025, such as audits, calibrations, and inter‐comparisons bring additional costs to the laboratory. In this aspect, Barradas and Sampaio [29], had shown that

The time necessary for implementing a quality management system depends basically on the size and complexity of the company/laboratory. It scales with the number of laboratories, testing services and R&D activities that the system will be applied. However, a functional The implementation of a quality management system in materials science laboratories based on the ISO/IEC 17025 standard must fulfil the requirements outlined in the standard. However, due to the particular characteristics of the work realized in such laboratories, some require‐ ments may be considered as key matters during the system implementation' these are as follows:


Some of these key matters should not only fulfil the standard requirements, but also provide the necessary flexibility for the activities carried out in those laboratories, especially those related to R&D management, like control of documents and records, for example. On the other hand, some of these topics are related to technical features specific for particular tests methods for materials characterization, such as evaluation of uncertainty and traceability. These last aspects may be considered independently of managerial aspects.

The success in implementing a functional and flexible quality management system in afore‐ mentioned science laboratories depends not only on the effective and efficient handling of these key factors, but also on the use of a suitable approach for implementing the system. This approach should use appropriate strategies and chronogram in order to facilitate the imple‐ mentation follow‐up and minimize the time and cost of the process.

One last, but not less important, consideration should be made on further improvement of quality assurance provided by materials science laboratories. As mentioned above in this chapter, materials science impacts innovation in several areas, both on the development of new materials and on the development and optimization of techniques for materials property characterization. In this context, it is very common that the use of new materials in commercial products give rise to safety concerns about the use of the product and its disposal in the environment [30, 31]. Some typical cases are, for example, the use of nanoparticles in cosmetics and pharmaceutical products [32], and use of new composites for engineering systems [33].

These issues are not directly related to the ISO/IEC 17025, but to specific international regula‐ tion for several products and to international standards used to perform materials and products tests. In this aspect, it is important for materials laboratories engaged in quality assurance to work proactively with national and international organizations responsible for standardization and/or regulation, collaborating on the development and optimization of standard procedures for materials characterization and proposition of technical criteria for utilization of materials for specific uses.

## **Author details**

Rodrigo S. Neves\* , Daniel P. Da Silva, Carlos E. C. Galhardo, Erlon H. M. Ferreira, Rafael M. Trommer and Jailton C. Damasceno

\*Address all correspondence to: rsneves@inmetro.gov.br

Materials Meteorology Division (Dimat)—National Institute of Meteorology, Quality and Technology (Inmetro), Rio de Janeiro, Brazil

## **References**

**•** Quality assurance of results and

**•** Evaluation of uncertainty of measurement.

42 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

utilization of materials for specific uses.

Rafael M. Trommer and Jailton C. Damasceno

Technology (Inmetro), Rio de Janeiro, Brazil

\*Address all correspondence to: rsneves@inmetro.gov.br

**Author details**

Rodrigo S. Neves\*

aspects may be considered independently of managerial aspects.

mentation follow‐up and minimize the time and cost of the process.

Some of these key matters should not only fulfil the standard requirements, but also provide the necessary flexibility for the activities carried out in those laboratories, especially those related to R&D management, like control of documents and records, for example. On the other hand, some of these topics are related to technical features specific for particular tests methods for materials characterization, such as evaluation of uncertainty and traceability. These last

The success in implementing a functional and flexible quality management system in afore‐ mentioned science laboratories depends not only on the effective and efficient handling of these key factors, but also on the use of a suitable approach for implementing the system. This approach should use appropriate strategies and chronogram in order to facilitate the imple‐

One last, but not less important, consideration should be made on further improvement of quality assurance provided by materials science laboratories. As mentioned above in this chapter, materials science impacts innovation in several areas, both on the development of new materials and on the development and optimization of techniques for materials property characterization. In this context, it is very common that the use of new materials in commercial products give rise to safety concerns about the use of the product and its disposal in the environment [30, 31]. Some typical cases are, for example, the use of nanoparticles in cosmetics and pharmaceutical products [32], and use of new composites for engineering systems [33].

These issues are not directly related to the ISO/IEC 17025, but to specific international regula‐ tion for several products and to international standards used to perform materials and products tests. In this aspect, it is important for materials laboratories engaged in quality assurance to work proactively with national and international organizations responsible for standardization and/or regulation, collaborating on the development and optimization of standard procedures for materials characterization and proposition of technical criteria for

, Daniel P. Da Silva, Carlos E. C. Galhardo, Erlon H. M. Ferreira,

Materials Meteorology Division (Dimat)—National Institute of Meteorology, Quality and


[Internet]. 08/2016 [Updated: 08/2016]. Available from: http://www.nata.com.au/nata/ nata-enews/66-nata-e-news/oct2015/1212-update-on-the-revision-of-isoiec-17025-2005-conformity-assessment-general-requirements-for-the-competence-oftesting-and-calibration-laboratories [Accessed: 09/19/2016]


microscopy and particle‐tracking analysis. Journal of Nanoparticles Research. 2014;16:2628. DOI: 10.1007/s11051‐014‐2628‐3

[27] Kestens V., Roebben G., Linsinger T. Development and validation of a differential scanning calorimetry purity determination method for polycyclic aromatic hydrocar‐ bons. Accreditation and Quality Assurance. 2010;15:269–281. DOI: 10.1007/ s00769‐010‐0632‐6

[Internet]. 08/2016 [Updated: 08/2016]. Available from: http://www.nata.com.au/nata/

iec-17025-2005-conformity-assessment-general-requirements-for-the-competence-of-

[14] Joint Committee for Guides in Metrology. International vocabulary of metrology‒basic and general concepts and associated terms (VIM). 3rd ed. JCGM/BIPM; 2012. 91 p. [15] Salas-Tellez J.A., Guardado-Perez J.A., Rosas-Gutierrez F., Mitani Y. A traceability scheme for materials metrology. Metrologia. 2010;47:S18–S22. DOI:

[16] Roebben G., Insinger T., Lamberty A., Emons H. Metrological traceability of the measured values of properties of engineering materials. Metrologia. 2010;47:S23–S31.

[17] Bureau International des Poids et Mesures. The BIPM key comparison database

[18] Versailles Project on Advanced Materials and Standards (VAMAS)—Formation and Objectives [Internet]. Available from: http://www.vamas.org/formation.html [Ac-

[19] NIST. Certificate—SRM 676a—Alumina Powder (Quantitative Analysis Powder Diffraction Standard) [Internet]. [Updated: 3/16/2016]. Available from: https://www-

[20] JCGM 100:2008 Evaluation of measurement data—Guide to the expression of uncer-

[21] Kindlarski E. Ishikawa Diagrams for problem solving. Quality Progress. 1984;17:26–30. [22] De Lange M.F., Vlugt T.J.H., Gascon, J., Kapteijn F. Adsorptive characterization of porous solids: error analysis guides the way. Microporous and Mesoporous Materials.

[23] NIST. Assessment of Uncertainties for the NIST 1016 mm Guarded-Hot-Plate Apparatus: Extended Analysis for Low-Density Fibrous-Glass Thermal Insulation. [Internet]. 2009. Available from: http://fire.nist.gov/bfrlpubs/build09/art003.html [Accessed:

[24] ISO 5725-1 Accuracy (trueness and precision) of measurement methods and results—

[25] De Temmerman P.-J., Lammertyn J., De Ketelaere B., Kestens V., Roebben, G., Verleysen E., Mast J. Measurement uncertainties of size, shape, and surface measurements using transmission electron microscopy of near-monodisperse, near-spherical nanoparticles. Journal of Nanoparticles Research. 2014;16:2177. DOI: 10.1007/s11051-013-2177-1 [26] De Temmerman P.-J., Verleysen E., Lammertyn J., Mast J. Size measurement uncertainties of near-monodisperse, near-spherical nanoparticles using transmission electron

[Internet]. Available from: http://kcdb.bipm.org/ [Accessed: 29/09/2016]

s.nist.gov/srmors/view\_cert.cfm?srm=676A [Accessed: 09/01/2016]

tainty in measurement. 1st ed. BIPM/JCGM; 2008. 120 p.

2014;200:199–215. DOI: 10.1016/j.micromeso.2014.08.048

Part 1: General principles and definitions. ISO; 1994.

nata-enews/66-nata-e-news/oct2015/1212-update-on-the-revision-of-iso-

testing-and-calibration-laboratories [Accessed: 09/19/2016]

44 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

10.1088/0026-1394/47/2/S02

cessed: 07/29/]

09/01/2016]

DOI: 10.1088/0026-1394/47/2/S03


**Chapter 3 Provisional chapter**

#### **Youden Two-Sample Method Youden Two-Sample Method**

Julia Martín, Nieves Velázquez and Agustin

Julia Martín, Nieves Velázquez and G. Asuero

Agustin G. Asuero

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66411

#### **Abstract**

The results obtained when testing materials, equipment and procedures are not generally identical. Factors that influence the magnitude of the results are not fully controllable. As such, the interpretation and analysis of results must take into account the variations caused by numerous and random unavoidable causes. Intercomparison exercises are considered of being of importance, as they do allow the examination of the analytical process and their generated results. Youden plot is particularly aimed at interlaboratory comparisons. The raw results provided by the participating laboratories are treated by a statistical method applied by the centre performing the trial. In order to materialize this, two similar materials with small differences in the concentration of the characteristics are required. The advantage of Youden analysis is its ability to separate the random errors with a minimum effort by participants in the design from the point of view of the analytical requirement. This book chapter illustrates the method that has been applied to elaborate on data covering a diverse scientific field: polyunsaturated fatty acids in fat and oils, total blood cholesterol and aspirin in pharmaceutical preparations. Finally, liquid chromatography with tandem mass spectrometry detector has been applied to the determination of an emerging contaminant, methylparaben (MeP), in surface waters.

**Keywords:** Youden plot, confidence ellipse, quality control

## **1. Introduction**

The main objective of quality systems when implanted in analytical laboratories is to ensure that the results obtained confirm to quality standards, in addition to them showing a level of harmonization [1–4] between obtained results.

In order to achieve this goal, quality assessment systems are implemented so as to allow the examination of the analytical process as well as of their results generated.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Quality assessment is coined as the systematic examination carried out by an entity to verify [5– 7] that it meets specified requirements (fitness for purpose). This is a generic concept that can be refined more by relating it to the specific set of activities planned and executed with the aim of ensuring that the activities involved in the quality control are done in a proper and efficient way.

The quality assessment involves the methodical and continuous contrast of the product, system or quality service. In the specific area of the laboratory, it refers to examination of systems and to analytical results generated both in terms of accuracy [8, 9] and representativeness.

Intercomparison exercises are framed in this context [10–13], establishing the procedure for design, organizing and gathering information from a set of laboratories working with the same samples that undergo an assessment of their results. An intercomparison exercise is based on acceptance by several laboratories to perform the same analysis. This analysis is carried out under the co-ordination of an organization. The purpose is to assess the quality of their work, to evaluate the method of measuring, to determine the property of a material (the content of an element or compound, etc.).

The main mission of the organization is to establish the objectives and the conditions regarding the participation of the laboratories [14–18] while ensuring the quality and stability of the sample under study. Those institutions are also responsible of dealing with the statistical treatment of obtained outcomes. The participating laboratories should, in turn, commit themselves to follow the conditions set by the organization, which may change depending on the type of executed exercise.

A very important aspect of intercomparison exercises is the selection of material to be used for the study. In that sense the type of matrix and the nature and range of values of the parameter (or parameters) under study must be defined.

It is worth mentioning at this point that not all materials are suitable to carry out a study of this type. It is essential that the material used is representative, homogeneous and stable. The organization is responsible for ensuring that the criteria mentioned above are met.

The preparation of the material must follow a series of stages, after which the material is packaged. It should, hence, be homogeneous and stable. The submitted sample must be properly identified and packaged in order to prevent breakage.

The sample is tagged accordingly so as to show the state in which the sample has been sent to the participating laboratory. Guidelines for sample preservation and handling, a description of the analytical methods to be applied as well as the report methods must also be included in the shipment.

Youden two-sample diagram, two-sample collaborative testing, two-sample plan, Youden plot or Youden analysis is particularly aimed at interlaboratory comparisons [19, 20]. The raw results provided by the participating laboratories are treated by a statistical method applied by the organizing centre of the trial. The Youden approach or z-score marks are some of the tools used for the treatment of the results. Finally, based on these statistical treatments, a statement is sent to the laboratories that have presented inaccurate results, as well as appropriate suggestions in order to improve their work.

A literature search has been carried out in order to validate current status with regards to the use of the Youden approach. The information gathered is presented in **Table 1** [1–129].

Performed literature review reveals that the Youden chart has been successfully used in agriculture, environmental chemistry, geochemistry, industry and medicine. The invention of the Youden diagram may be regarded as setting a landmark in quality control in clinical chemistry [21].

Quality assessment is coined as the systematic examination carried out by an entity to verify [5– 7] that it meets specified requirements (fitness for purpose). This is a generic concept that can be refined more by relating it to the specific set of activities planned and executed with the aim of ensuring that the activities involved in the quality control are done in a proper and efficient way.

The quality assessment involves the methodical and continuous contrast of the product, system or quality service. In the specific area of the laboratory, it refers to examination of systems and to analytical results generated both in terms of accuracy [8, 9] and representativeness.

Intercomparison exercises are framed in this context [10–13], establishing the procedure for design, organizing and gathering information from a set of laboratories working with the same samples that undergo an assessment of their results. An intercomparison exercise is based on acceptance by several laboratories to perform the same analysis. This analysis is carried out under the co-ordination of an organization. The purpose is to assess the quality of their work, to evaluate the method of measuring, to determine the property of a material (the

The main mission of the organization is to establish the objectives and the conditions regarding the participation of the laboratories [14–18] while ensuring the quality and stability of the sample under study. Those institutions are also responsible of dealing with the statistical treatment of obtained outcomes. The participating laboratories should, in turn, commit themselves to follow the conditions set by the organization, which may change depending on the type of executed exercise. A very important aspect of intercomparison exercises is the selection of material to be used for the study. In that sense the type of matrix and the nature and range of values of the parameter

It is worth mentioning at this point that not all materials are suitable to carry out a study of this type. It is essential that the material used is representative, homogeneous and stable. The

The preparation of the material must follow a series of stages, after which the material is packaged. It should, hence, be homogeneous and stable. The submitted sample must be properly

The sample is tagged accordingly so as to show the state in which the sample has been sent to the participating laboratory. Guidelines for sample preservation and handling, a description of the analytical methods to be applied as well as the report methods must also be included in the shipment.

Youden two-sample diagram, two-sample collaborative testing, two-sample plan, Youden plot or Youden analysis is particularly aimed at interlaboratory comparisons [19, 20]. The raw results provided by the participating laboratories are treated by a statistical method applied by the organizing centre of the trial. The Youden approach or z-score marks are some of the tools used for the treatment of the results. Finally, based on these statistical treatments, a statement is sent to the laboratories that have presented inaccurate results, as well as appro-

A literature search has been carried out in order to validate current status with regards to the use of the Youden approach. The information gathered is presented in **Table 1** [1–129].

organization is responsible for ensuring that the criteria mentioned above are met.

content of an element or compound, etc.).

48 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

(or parameters) under study must be defined.

identified and packaged in order to prevent breakage.

priate suggestions in order to improve their work.

The performance and evaluation of interlaboratory programmes by Youden's method are suited to laboratory monitoring and allow to obtain information concerning both precision and systematic errors from analytical results without much effort.


and Singapore



**Content Reference** Collaborative test on the count of *Escherichia coli* [35]

[36]

[37]

[39]

[40]

[41]

[42]

[43]

[45]

[46]

[47]

[48]

[49]

[50]

[52]

[1]

[11]

Proficiency test for the determination of heavy metals in mineral feed. The importance of correctly

Performances of analytical methods for atmospheric deposition and soil analysis assessed through

Second interlaboratory exercise on non-steroidal anti-inflammatory drug analysis in environmental

HIV external quality assessment (EQA) results by the KCDC from the 17 HIV testing laboratories that also

Statics and chemometrics for analytical chemistry [7] A proficiency testing scheme was developed for a limited number of analytical laboratories participating

Proficiency test for heavy metals in feed and food in Europe [44]

A multilaboratory proficiency testing programme was conducted by the National Accreditation Board for Testing and Calibration Laboratories (India) and coordinated by the Institute of Pesticide Formulation Technology. This programme was conducted to compare the performance of individual laboratories in the area of pesticide formulation (Chlorpyrifos 20 EC) analysis. A total of 24 laboratories in India participated

Proficiency testing for the determination of pesticides in mango pulp: a view of the employed

Investigations for the improvement of the measurement of volatile organic compounds from floor coverings within the health-related evaluation of construction products: application of the Youden method

Characterization of candidate reference materials for bone lead via interlaboratory study and double

Implementation and methodology of an interlaboratory system that ensures the quality of glassware

Description and results of the 2005 interlaboratory comparison exercise for trace elements in marine mammals. Two quality control materials derived from fresh-frozen marine mammal livers were produced

An updated liquid chromatographic assay for the determination of glyphosate in technical material and

Collaborative studies for quantitative chemical analytical methods [51]

Youden method applied to the external quality control of semen analysis in Germany [53]

Brief note on the Youden method [54] Interlaboratory comparison by means of method performance precision and bias studies and proficiency testing schemes are described. The set-up of the experiments and the evaluation of the data by means of

Quality assurance in analytical chemistry application in the environmental, food and materials analysis,

chromatographic techniques and the evaluation of laboratories' performance

and characterized at the NIST and were then distributed to over 30 laboratories

Evaluation tools to understand statistical methods related to the *z*-score for use in proficiency testing by

Evaluation of learning outcomes in quantitative analysis lab using Youden plots [38] State of the art with respect to the selection and use of proficiency testing schemes and the interpretation

selecting the certified reference materials during method validation

50 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

of results and evaluations given in proficiency testing schemes

performed HIV-1 western blot testing of the 585 laboratories

interlaboratory comparisons

intercomparison exercises

in the analysis of natural water in Israel

isotope dilution mass spectrometry

calibration and use in a large laboratory

biotechnology and medical engineering

graphical and statistical methods are considered

formulations: application of the Youden method

aqueous samples



**Table 1.** Some published papers dealing with the Youden approach.

**Content Reference**

[73]

[74]

[78]

[9]

[88]

[16]

[93]

Intercomparison exercise on the determination of organochlorine compounds and petroleum

52 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Application and improvement of the Youden analysis in the intercomparison between flowmeter

Assessment of overall accuracy of lead isotope ratios determined by inductively coupled plasma mass

World Health Organization international intercalibration study on dioxins and furans in human milk

interlaboratory studies and a simplified approach to the statistical analysis of collaborative study results

Robust statistic and functional relationship estimation for comparing the bias of analytical procedures

Guidelines for the development of standard methods of collaborative study: organization of

Analytical quality assurance. A review [89] Multiway analysis of variance for the interpretation of interlaboratory studies [90] External quality control study on the reliability of current histamine determinations in European laboratories [91]

Classic paper reprint. The collaborative test of Youden [92]

Bias-free adjustment of analytical methods to laboratory samples in routine analytical procedures [94] Protocol for the design, conducting and interpretation of method-performance studies [3] Exchange of comments on a new technique in chemical assay calculations [95]

spectrometry using batch quality control and the Youden two-sample method

Statistical model assumptions upon which the procedure is based. Provides validity tests for several of these assumptions, explains conditions under which Youden is not consistent with precision estimate and indicates when precision estimates based on the procedure should be interpreted with caution or should not be used

Intralaboratory testing of method accuracy from recovery assays [8] Application of the Youden method to the mass fraction Youden protein fodder [75] Succinct description of the two-sample Youden method [19] Performances of analytical methods for freshwater analysis assessed through intercomparison exercises [76] Proposed guidelines for the internal quality control of analytical results inthe medical laboratories [77]

Basic of interlaboratory studies: the trends in the new ISO 5725 standard edition [79] Round-robin study of performance evaluation soils vapor-fortified with volatile organic compounds [80] Protocol for the design, conducting and interpretation of collaborative studies [2] Application of the Youden method to acid rain analites [81] A bivariate control chart for paired measurements [82] Polystyrene film as a standard for testing FT-IR spectrometers [83] Graphical diagnosis of interlaboratory quality control data for surface water samples [84] Nomenclature of interlaboratory analytical studies [85] Basic method for the determination of repeatability and reproducibility of a standard measurement method [86] Reviews on the life and work of Youden [87] Quality control in analytical chemistry [6]

hydrocarbons in algae

calibration facilities

and blood

over extended concentration ranges

## **2. Literature review: the Youden plot**

W. J. Youden (1900–1973) was a physical chemist during the first third of this life, who turned into a statistician later, employed by the National Bureau of Standards (NBS) (now National Institute of Standards and Technology, NIST) from 1948 until his death in 1971. One of his more memorable sentences states [22, p. 12] that "The best way to find out about some of the difficulties in making measurements is to make measurements" [22].

He approached interlaboratory testing as a means of uncovering biases in measurement processes, and the so-called Youden plot has become an accepted design and analysis technique throughout the world for comparing precision and bias amongst laboratories. Youden suggested in 1959 a very simple graphical procedure for plotting results obtained by different laboratories [23–25]. Work in graphical methods, which began with the Youden plot, continues today, notably in recent works of NIST chemists.

The above is also referred to as two-sample collaborative testing, two-sample diagram, twosample plan or Youden plot.

The method focuses on intercomparison exercises. The main characteristic is its ability to separate the systematic and random errors with minimal effort on the part of the participants.

The method is implemented as follows:

Two nearly identical samples are prepared, divided and sent to each of the participating laboratories, as recommended by Youden. A scatter plot is drawn in which the x-axis indicates one of the reported values and the y-axis the other. The scale units are the same along each axis. Each pair of results, corresponding to a given laboratory, is a point in the Youden plot (see **Figure 1**).

The points will cluster in a circular pattern whose centre is the mean values for the two samples.

Once the results are represented in the plot, they are divided into four quadrants, which are identified as (+, +), (−, +), (−, −) and (+, −). When any laboratory's result exceeds the mean achieved for all laboratories, a plus sign is used, a minus sign indicates a value smaller than the mean. If the variation in results is dominated by random errors, it would be expected that the points fall randomly distributed in all quadrants, with similar number of points in each quadrant. When systematic errors are significantly larger than random errors, then the points occur primarily in the (+, +) and the (−, −) quadrants, forming an elliptical pattern around a line bisecting these quadrants at a 45° angle.

The plot is an effective method to qualitatively evaluate the results and the capabilities of the proposed method. As can be seen in **Figure 2**, the length of a perpendicular line from any point to the 45° line is proportional to the contribution of random error on a given laboratory's results (red arrow). The distance from the intersection of the axes (mean values for samples X and Y) to the perpendicular projection of a point on the 45° line is proportional to the laboratory's systematic error (green arrow).

**2. Literature review: the Youden plot**

54 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

ues today, notably in recent works of NIST chemists.

sample plan or Youden plot.

(see **Figure 1**).

samples.

The method is implemented as follows:

line bisecting these quadrants at a 45° angle.

tory's systematic error (green arrow).

W. J. Youden (1900–1973) was a physical chemist during the first third of this life, who turned into a statistician later, employed by the National Bureau of Standards (NBS) (now National Institute of Standards and Technology, NIST) from 1948 until his death in 1971. One of his more memorable sentences states [22, p. 12] that "The best way to find out about some of the

He approached interlaboratory testing as a means of uncovering biases in measurement processes, and the so-called Youden plot has become an accepted design and analysis technique throughout the world for comparing precision and bias amongst laboratories. Youden suggested in 1959 a very simple graphical procedure for plotting results obtained by different laboratories [23–25]. Work in graphical methods, which began with the Youden plot, contin-

The above is also referred to as two-sample collaborative testing, two-sample diagram, two-

The method focuses on intercomparison exercises. The main characteristic is its ability to separate the systematic and random errors with minimal effort on the part of the participants.

Two nearly identical samples are prepared, divided and sent to each of the participating laboratories, as recommended by Youden. A scatter plot is drawn in which the x-axis indicates one of the reported values and the y-axis the other. The scale units are the same along each axis. Each pair of results, corresponding to a given laboratory, is a point in the Youden plot

The points will cluster in a circular pattern whose centre is the mean values for the two

Once the results are represented in the plot, they are divided into four quadrants, which are identified as (+, +), (−, +), (−, −) and (+, −). When any laboratory's result exceeds the mean achieved for all laboratories, a plus sign is used, a minus sign indicates a value smaller than the mean. If the variation in results is dominated by random errors, it would be expected that the points fall randomly distributed in all quadrants, with similar number of points in each quadrant. When systematic errors are significantly larger than random errors, then the points occur primarily in the (+, +) and the (−, −) quadrants, forming an elliptical pattern around a

The plot is an effective method to qualitatively evaluate the results and the capabilities of the proposed method. As can be seen in **Figure 2**, the length of a perpendicular line from any point to the 45° line is proportional to the contribution of random error on a given laboratory's results (red arrow). The distance from the intersection of the axes (mean values for samples X and Y) to the perpendicular projection of a point on the 45° line is proportional to the labora-

difficulties in making measurements is to make measurements" [22].

**Figure 1.** Typical Youden plots when (a) random errors are significantly larger than systematic errors due to the analysts and (b) when systematic errors due to the analysts are significantly larger than the random errors [126].

**Figure 2.** Relationship between the result for a single laboratory (in blue) and the contribution of random error (red arrow) and the contribution from the laboratory's systematic error (green arrow) [126].

An ideal standard method is linked with small random and systematic errors characterizing a circular compact cluster of points.

The Youden plot is a special case of the bivariate control chart to evaluate the performance of several laboratories, and the idea behind is the principal component analysis [26].

In 1974, the Youden plot was extended by Mandel and Lashof by using an ellipse instead of a circle [27]. In Youden's original method, the concentration of the analyte in the two materials was nearly the same, so that the repeatability as well as the laboratory biases would be the same for two materials [28].

Mandel and Lashof investigate the situation where two samples do not have a similar concentration so that random and systematic errors are no longer necessarily the same for both methods. They showed that in all cases, the points in the plot fall within an elongated ellipse. When Youden's original plot (two similar samples) is applied to, then the major axis forms a 45° angle. In retrospect, when samples are not similar to one another, different angles may be obtained. Their paper contains a procedure to decide whether lab bias occurs or not and also contains an estimate of all variance components.

The confidence ellipse has been proposed in ISO 13528:2005 to indicate anomalies in the between- and the within-laboratory errors in qualitative terms. For laboratory monitoring, interlaboratory tests performed according to ISO 5725-2 require much effort, especially because a large volume of samples must be provided by the organizer for *K* = 4 repeated analyses per laboratory. It is worth noting at this point that this is only suited for process standardization.

As already mentioned (see pp. 3–4), the performance and evaluation of interlaboratory programmes by Youden's method are recommended above all for laboratory monitoring. It allows to obtain information concerning both precision and systematic errors from analytical results without much effort. In addition to the above, Youden's method requires less effort for organizers and participants alike. It equally showcases a simple evaluation meaning that a potential manipulation is less likely.

Youden's method has been recommended in modern statistical manuals, procedures and protocols [29, 30] as well as recent papers, reports and government agencies, as depicted in **Table 1** [31–34].

This study discusses the Youden method and elaborates its applications in a number of diverse areas.

The experimental systems selected first for gaining experience and training in the application of the method have been the determination of polyunsaturated fatty acids in fats and oils [35], total blood cholesterol [36] and aspirin in pharmaceutical preparations [37], i.e. food and clinical and pharmaceutical applications, respectively.

Finally, a detailed procedure for the determination of methylparaben in surface waters, of special relevance nowadays in the environmental field, has been developed by using liquid chromatography-mass spectrometry.

At last, the confidence ellipse as proposed by ISO 13528:2005 will be described, and a practical case of concentrations of antibodies for two similar allergens is used as an aid to interpret the plot.

#### **2.1. Youden plot development**

The Youden plot is a special case of the bivariate control chart to evaluate the performance of

In 1974, the Youden plot was extended by Mandel and Lashof by using an ellipse instead of a circle [27]. In Youden's original method, the concentration of the analyte in the two materials was nearly the same, so that the repeatability as well as the laboratory biases would be the

Mandel and Lashof investigate the situation where two samples do not have a similar concentration so that random and systematic errors are no longer necessarily the same for both methods. They showed that in all cases, the points in the plot fall within an elongated ellipse. When Youden's original plot (two similar samples) is applied to, then the major axis forms a 45° angle. In retrospect, when samples are not similar to one another, different angles may be obtained. Their paper contains a procedure to decide whether lab bias occurs or not and also

The confidence ellipse has been proposed in ISO 13528:2005 to indicate anomalies in the between- and the within-laboratory errors in qualitative terms. For laboratory monitoring, interlaboratory tests performed according to ISO 5725-2 require much effort, especially because a large volume of samples must be provided by the organizer for *K* = 4 repeated analyses per laboratory. It is worth noting at this point that this is only suited for process

As already mentioned (see pp. 3–4), the performance and evaluation of interlaboratory programmes by Youden's method are recommended above all for laboratory monitoring. It allows to obtain information concerning both precision and systematic errors from analytical results without much effort. In addition to the above, Youden's method requires less effort for organizers and participants alike. It equally showcases a simple evaluation meaning that a

Youden's method has been recommended in modern statistical manuals, procedures and protocols [29, 30] as well as recent papers, reports and government agencies, as depicted in

This study discusses the Youden method and elaborates its applications in a number of

The experimental systems selected first for gaining experience and training in the application of the method have been the determination of polyunsaturated fatty acids in fats and oils [35], total blood cholesterol [36] and aspirin in pharmaceutical preparations [37], i.e. food and clini-

Finally, a detailed procedure for the determination of methylparaben in surface waters, of special relevance nowadays in the environmental field, has been developed by using liquid

At last, the confidence ellipse as proposed by ISO 13528:2005 will be described, and a practical case of concentrations of antibodies for two similar allergens is used as an aid to interpret the plot.

several laboratories, and the idea behind is the principal component analysis [26].

same for two materials [28].

standardization.

**Table 1** [31–34].

diverse areas.

potential manipulation is less likely.

cal and pharmaceutical applications, respectively.

chromatography-mass spectrometry.

contains an estimate of all variance components.

56 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

The Youden plot is developed as follows:


$$S\_D^{-2} = \frac{\Sigma (Di - \text{Dmean})^2}{2(l-1)}\tag{1}$$

where l is the number of laboratories and the factor of 2 is the result of using two values to determine D<sup>i</sup> .

5. In the same way, the total, T, of each laboratory's results (T<sup>i</sup> = X<sup>i</sup> + Y<sup>i</sup> ) contains contributions from both random error and twice the laboratory's systematic error:

$$
\sigma\_{\rm tot}^2 = \sigma\_{\rm rand}^2 + \mathcal{D}\sigma\_{\rm syst}^2 \tag{2}
$$

6. The standard deviation of the totals, S<sup>T</sup> , provides an estimate for σtot:

$$S\_T^{-2} = \frac{\Sigma(\text{Ti} - \text{Time})^2}{2(l-1)}\tag{3}$$

Again, the factor of 2 in the denominator is the result of using two values to determine T<sup>i</sup> .


$$
\sigma\_{\rm SSS}^{-2} = \frac{S\_T^{-2} - S\_D^{-2}}{2} \tag{4}
$$

9. Calculating the radius of the confidence circle:

$$\mathbf{R} = \mathbf{SD} \cdot \mathbf{b} \tag{5}$$

$$b = -2\ln\left(1 - \frac{P}{100}\right) \tag{6}$$

$$P = 100\left(1 - \exp\left(\frac{-b^2}{2}\right)\right) \tag{7}$$

where % P is the percentage of selected confidence level (usually 95%).

10. Draw a circle with radius R and centroid (X-mean, Y-mean). The laboratories falling outside the 95% circle are said to provide biased results. The radius of the circle is based on a multiple of SD, depending on the desired percentage of observations anticipated to fall within a bivariate normal distribution. A circle whose radius is a multiple of SD = (2.5; 3) represents the smallest circle that can be contained in almost every point, in the absence of bias.

#### **2.2. An alternative approximation: the** *Z***-score**

An alternative to Youden Plot is the punctuation *Z*-Score. This value is used to "score" a parameter in a particular round of a laboratory's participation. This is done by means of the following calculations:


2. Calculate the total error \(\varepsilon\_{\parallel}\) for each laboratory:

$$\varepsilon\_{\perp} = \sqrt{(\underline{\mathbf{x}}\_{i} - \underline{\mathbf{x}}\_{\text{Md}})^{2} + (\underline{\mathbf{y}}\_{i} - \underline{\mathbf{y}}\_{\text{Md}})^{2}}\tag{8}$$

3. Calculate the systematic error component (C<sup>s</sup> ) as

\*\*3.\*\* Calculate the systematic error component \(C\_s\) as

$$C\_s = \frac{(\mathbf{x}\_i - \mathbf{x}\_{\text{Ma}}) + (y\_i - y\_{\text{Ma}})}{\sqrt{2}}\tag{9}$$

4. Calculate the random error component (CR) as

\*\*4.\*\* \*\*C\*\*:\*\* \*\*c\*\*\*\* \*\*me \*\*r\*\*\*\* \*\*n\*\* \*\*corport\*\* \*\*corport\*\* \*\*( $\leftarrow\_{\kappa}$ )\*\* as\*\*

$$C\_{\kappa} = \sqrt{\left(x\_{1} - \left(\frac{C\_{s}}{\sqrt{2}}\right) + x\_{\text{Mo}}\right)^{2} + \left(y\_{1} - \left(\frac{C\_{s}}{\sqrt{2}}\right) + y\_{\text{Mo}}\right)^{2}}\tag{10}$$

5. Systematic and random components calculation required so that their sum is equal to the magnitude of the total error:

Systematic error:

$$\mathbf{e}\_{\rm s} = \frac{\mathbf{C}\_{\rm s}}{|\mathbf{C}\_{\rm s}| + \mathbf{C}\_{\rm R}} \mathbf{e}\_{\rm r} \tag{11}$$

Random error:

$$\mathbf{e}\_{\mathrm{k}} = \frac{\mathbf{C}\_{\mathrm{k}}}{|\mathbf{C}\_{\mathrm{s}}| + \mathbf{C}\_{\mathrm{k}}} \mathbf{e}\_{T} \tag{12}$$

Thus,

$$E\_T = \left\lfloor \varepsilon\_\xi \right\rfloor + \varepsilon\_\aleph \tag{13}$$

6. Calculate the typical deviation:

*P* = 100(1 − exp(

58 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**2.2. An alternative approximation: the** *Z***-score**

2. Calculate the total error (*εT*) for each laboratory:

3. Calculate the systematic error component (C<sup>s</sup>

4. Calculate the random error component (CR) as

<sup>ԑ</sup>*<sup>S</sup>* <sup>=</sup> *<sup>C</sup>* \_\_\_\_\_\_\_ *<sup>s</sup>*

<sup>ԑ</sup>*<sup>R</sup>* <sup>=</sup> *<sup>C</sup>* \_\_\_\_\_\_\_ *<sup>R</sup>*

*Cs* <sup>=</sup> (*xi* <sup>−</sup> *xMed*) <sup>+</sup> (*yi* <sup>−</sup> *<sup>y</sup>*Me)

of bias.

following calculations:

1. Calculate the median of X and Y.

ԑ*<sup>T</sup>* = √

*CR* <sup>=</sup> <sup>√</sup>

magnitude of the total error:

Systematic error:

Random error:

Thus,

where % P is the percentage of selected confidence level (usually 95%).

−*b*<sup>2</sup> \_\_\_

10. Draw a circle with radius R and centroid (X-mean, Y-mean). The laboratories falling outside the 95% circle are said to provide biased results. The radius of the circle is based on a multiple of SD, depending on the desired percentage of observations anticipated to fall within a bivariate normal distribution. A circle whose radius is a multiple of SD = (2.5; 3) represents the smallest circle that can be contained in almost every point, in the absence

An alternative to Youden Plot is the punctuation *Z*-Score. This value is used to "score" a parameter in a particular round of a laboratory's participation. This is done by means of the

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

) as

 \_\_\_\_\_\_\_\_\_\_\_\_\_ √ \_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ (*x*<sup>1</sup> <sup>−</sup> ( *<sup>C</sup>*\_\_\_*<sup>S</sup>*

+ (*y*<sup>1</sup> − (

*C*\_\_\_*S* √ \_\_ <sup>2</sup>) <sup>+</sup> *<sup>y</sup>*Me)

2

5. Systematic and random components calculation required so that their sum is equal to the



*ET* = | ԑ*S*| + ԑ*<sup>R</sup>* (13)

<sup>2</sup> + (*yi* − *y*Me)

(*xi* − *x*Me)

√ \_\_ <sup>2</sup>) <sup>+</sup> *<sup>x</sup>*Me) <sup>2</sup> )) (7)

2 (8)

(10)

<sup>2</sup> (9)

ԑ*<sup>T</sup>* (11)

ԑ*<sup>T</sup>* (12)

2

$$
\sigma = \sqrt{\frac{\sum\_{i=1}^{n} \varepsilon\_{ii}^{-2}}{n-1}} \tag{14}
$$

where n is the number of laboratories.

7. Finally, the z-score is calculated as

$$\mathbf{z} = \frac{\mathbf{x}\_i - \mathbf{x}\_{\text{Mo}}}{\sigma} \tag{15}$$

The results are classified as


### **3. Application to experimental systems**

#### **3.1. Determination of polyunsaturated fatty acids in fats and oils**

To illustrate the procedure, data from the interlaboratory study of a method for determining polyunsaturated fatty acids (PUFA) in fats and oils is used. The procedure consisted of saponifying a sample, treating it with an enzyme and measuring the absorbing product at 234 nm. Palm oil, corn oil and three hydrogenated blends were used in the study. One blend of hydrogenated oil was separated into two parts and designated as samples X and Y, respectively. Random subsamples from the two samples were analysed in each of the 17 laboratories as blind duplicates. The test incorporated an official Food and Drug Administration (FDA) method and a slightly modified one using boron trifluoride-methanol (BF). The aim in this case was to identify the laboratories that have higher quality results.

The results for FDA method are shown in **Table 2**. The data from laboratory 1 were rejected because inaccurate results were proportioned; those from laboratory 11 are listed but were not used (because their results were beyond the ones shown by others (8.2 g/100 g for sample X and 26.3 g/100 g for sample Y) in the calculations.

The vertical line at 28.6 g/100 g is the average value for sample X, and the average value for sample Y is shown by the horizontal line at 28.2 g/100 g. To estimate σrand and σsyst, the values for D<sup>i</sup> and Ti are calculated first. Next, the standard deviations for the differences, SD, and the totals, S<sup>T</sup> , are computed using Eqs. (1) and (3), yielding SD = 1.53 and ST = 3.11. To determine if the systematic errors between the laboratories are significant, the F-test is applied so as to compare ST and SD.

Because the F-ratio (4.141) is larger than F(0.05,14,14), which is 2.484, it is concluded that the systematic errors between the analysts are significant at the 95% confidence level, which is estimated using Eq. (4) giving 3.67.


**Table 2.** Determination of cis, cis-PUFA in blind duplicate samples by two methods (g trilinolein/100 g sample).

The results are plotted in **Figure 3**. The latter reveals that laboratories 11 (aberrant result), 12, 13, 14, 15 and 16 are outside the 95% circle, indicating high systematic errors.

The results for the BF method are shown in **Table 2**.

Again, the data from laboratory 1 were rejected because of a mistake; those from laboratories 11 and 12 are listed but were not used (10.3 and 10.5 g/100 g, respectively, for sample X and 29.6 and 10.1 g/100 g, respectively, for sample Y), as they lie beyond the set limit [35] in the calculations.

Again, the F-ratio (3.268) is larger than F(0.05,13,13), which is 2.577, so it is determined that the systematic errors between the laboratories are significant at the 95% confidence level. All the results are plotted in **Figure 4**. By observing the latter, one may observe that the laboratories 11, 12 (aberrants) and 2 and 13 are outside the 95% circle and are displaced far from the cluster of the others.

Notice that in both methods, about half the points lie above, and about half lie below the horizontal lines through the two means. Likewise, the vertical lines also separate the laboratories into equal groups, as do the 45° lines. However, in neither plot are the results equally distributed amongst the four quadrants; there are more in the upper right and lower left quadrants than in the upper left and lower right. Dispersion along the 45° line indicates that laboratories are high or low on both samples, while dispersion at right angles to the 45° line indicates a lack of agreement between results from the same laboratory.

**Figure 3.** Youden plot (determination of PUFA, FDA method).

The results are plotted in **Figure 3**. The latter reveals that laboratories 11 (aberrant result), 12,

**Table 2.** Determination of cis, cis-PUFA in blind duplicate samples by two methods (g trilinolein/100 g sample).

Again, the data from laboratory 1 were rejected because of a mistake; those from laboratories 11 and 12 are listed but were not used (10.3 and 10.5 g/100 g, respectively, for sample X and 29.6 and 10.1 g/100 g, respectively, for sample Y), as they lie beyond the set limit [35] in the calculations. Again, the F-ratio (3.268) is larger than F(0.05,13,13), which is 2.577, so it is determined that the systematic errors between the laboratories are significant at the 95% confidence level. All the results are plotted in **Figure 4**. By observing the latter, one may observe that the laboratories 11, 12 (aberrants) and 2 and 13 are outside the 95% circle and are displaced far from the

Notice that in both methods, about half the points lie above, and about half lie below the horizontal lines through the two means. Likewise, the vertical lines also separate the laboratories into equal groups, as do the 45° lines. However, in neither plot are the results equally distributed amongst the four quadrants; there are more in the upper right and lower left quadrants than in the upper left and lower right. Dispersion along the 45° line indicates that laboratories are high or low on both samples, while dispersion at right angles to the 45° line indicates a

13, 14, 15 and 16 are outside the 95% circle, indicating high systematic errors.

**Laboratory FDA method Laboratory BF method**

60 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

 26.1 28.5 2 24.9 29.4 29.6 28.6 3 30.3 29.4 29.2 26.8 4 29.1 31.7 29.5 26.9 5 31.4 29.3 30.3 30.8 6 29.1 30.4 27.5 25.9 7 26.6 26.6 25.8 26.9 8 30.0 30.7 30.0 28.0 9 29.5 29.7 29.0 25.0 10 28.3 29.3 11\* 8.20 26.3 11\* 10.3 29.6 31.3 32.0 12\* 10.5 10.1 24.7 24.8 13 25.3 24.8 24.3 25.9 14 26.3 28.6 31.0 31.3 15 31.4 30.3 28.2 32.3 16 28.0 28.0 31.8 29.9 17 29.6 27.0

**Sample X Sample Y Sample X Sample Y**

The results for the BF method are shown in **Table 2**.

lack of agreement between results from the same laboratory.

cluster of the others.

\*Not included in mean.

**Figure 4.** Youden plot (determination of PUFA, BF method).

If there were no systematic variations amongst laboratories, the pattern of points would be expected to be circular. The greater the systematic variations, the more elliptical the pattern will become.

The results obtained by the laboratories may now be compared by using the two methods as follows:


#### **3.2. Determination of cholesterol levels in the blood**

As part of a collaborative study to assess a new method that allows the determination of the total amount of cholesterol in the blood, two samples and the instructions to analyse each sample are sent to ten laboratories [36].

**Table 3** shows the results obtained in mg total cholesterol per 100 mL of serum.

**Figure 5** provides a two-sample plot of the results. The clustering of points suggests that the systematic errors of the analysts are significant. Two laboratories (1, 5) are outside the limit on the plot and widely separated from each other.

The vertical line at 248.9 mg/100 mL is the mean value for sample X, whereas the horizontal line at 246.5 mg/100 mL corresponds to the mean value of sample Y. To estimate σrand and σsyst, the values for D<sup>i</sup> and Ti are calculated first, followed by the standard deviations.

Because the F-ratio (2.530) is lower than F(0.05,9,9), which is 3.179, it is concluded that the systematic errors between the analysts are insignificant at the 95% confidence level.

If the true values for both samples are known, it is possible to test the presence of systematic errors. When there are no systematic method errors, the sum of the true values, μtot, for samples X and Y is equal to

$$
\mu \text{tot} = \mu \mathcal{X} + \mu \mathcal{Y} \tag{16}
$$

should fall within the confidence interval around *T*. A two-tailed *t*-test of the following null and alternate hypotheses is applied:

$$\text{H0: T = \mu} \text{tot } \text{HA: T \neq \mu} \text{tot} \text{ } \text{s}$$


**Table 3.** Determination of cholesterol in serum (mg cholesterol/100 mL of serum).

If there were no systematic variations amongst laboratories, the pattern of points would be expected to be circular. The greater the systematic variations, the more elliptical the pattern

The results obtained by the laboratories may now be compared by using the two methods as

 - Three laboratories (3, 6 and 9) are within the 95% circle and near each other on both plots. - Five laboratories (4, 5, 8, 10 and 17) are within the 95% circle on both plots but widely

As part of a collaborative study to assess a new method that allows the determination of the total amount of cholesterol in the blood, two samples and the instructions to analyse each

**Figure 5** provides a two-sample plot of the results. The clustering of points suggests that the systematic errors of the analysts are significant. Two laboratories (1, 5) are outside the limit on

The vertical line at 248.9 mg/100 mL is the mean value for sample X, whereas the horizontal line at 246.5 mg/100 mL corresponds to the mean value of sample Y. To estimate σrand and σsyst,

Because the F-ratio (2.530) is lower than F(0.05,9,9), which is 3.179, it is concluded that the

If the true values for both samples are known, it is possible to test the presence of systematic errors. When there are no systematic method errors, the sum of the true values, μtot, for

should fall within the confidence interval around *T*. A two-tailed *t*-test of the following null

H0: T = μtot HA: T ≠ μtot

systematic errors between the analysts are insignificant at the 95% confidence level.

are calculated first, followed by the standard deviations.

μtot = μX + μY (16)

**Table 3** shows the results obtained in mg total cholesterol per 100 mL of serum.





**3.2. Determination of cholesterol levels in the blood**

62 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

sample are sent to ten laboratories [36].

and Ti

the values for D<sup>i</sup>

samples X and Y is equal to

and alternate hypotheses is applied:

the plot and widely separated from each other.

will become.

separated from each other.

follows:

**Figure 5.** Youden plot (determination of cholesterol).

This occurs so as to determine if there is evidence for a systematic error in the method. The test statistic, *t*exp, is

$$t\_{\rm cap} = \frac{\left| \vec{T} - \mu\_{\rm tot} \right| \sqrt{\eta}}{S\_r \sqrt{\mathcal{Z}}} \tag{17}$$

with *n*−1 degrees of freedom. The √ \_\_ 2 in is included in the denominator because ST underestimates the standard deviation when comparing *T* to μtot.

Because this value for *t*exp is smaller than the critical value of 2.26 for *t*(0.05, 9), there is no evidence for a systematic error in the method at the 95% confidence level.

#### **3.3. Determination of aspirin in pharmaceutical preparations**

The results of determinations, made in ten laboratories, on two similar aspirin preparations are given in **Table 4** [37].


**Table 4.** Weight content (%) of aspirin in pharmaceutical preparations [37].

The analysis of data concerning materials X and Y reveals the following:


The average values for the materials (50.054% for X and 52.068% for Y) are used for the centroid. The results are plotted in **Figure 6** where it can be deduced that the data pointed on the diagram fall in either the first or the third quadrant (line X = Y). This is a consequence of the fact that laboratories which obtained "high" values for material X also obtained "high" values for material Y. The opposite is also true; laboratories reporting "low" values for X also reported "low" values for Y.

**Figure 6.** Youden plot (determination of aspirin).

This occurs so as to determine if there is evidence for a systematic error in the method. The

Because this value for *t*exp is smaller than the critical value of 2.26 for *t*(0.05, 9), there is no

The results of determinations, made in ten laboratories, on two similar aspirin preparations

(17)

2 in is included in the denominator because ST underesti-

exp <sup>=</sup> <sup>|</sup>*<sup>T</sup>* ´ − *μ*tot| √ \_\_ *<sup>n</sup>* \_\_\_\_\_\_\_\_ *ST* √ \_\_ 2

\_\_

evidence for a systematic error in the method at the 95% confidence level.

**Laboratory Sample X Sample Y** 50.45 52.55 49.89 52.00 49.60 51.70 50.26 52.11 49.78 51.79 49.92 51.81 50.22 52.35 50.40 52.26 50.17 52.24 49.85 51.87

The analysis of data concerning materials X and Y reveals the following:

**Table 4.** Weight content (%) of aspirin in pharmaceutical preparations [37].


The average values for the materials (50.054% for X and 52.068% for Y) are used for the centroid. The results are plotted in **Figure 6** where it can be deduced that the data pointed on the diagram fall in either the first or the third quadrant (line X = Y). This is a consequence of the fact that laboratories which obtained "high" values for material X also obtained "high" values for material Y. The opposite is also true; laboratories reporting "low" values for X also


materials is approximately the same.

reported "low" values for Y.

mates the standard deviation when comparing *T* to μtot.

64 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**3.3. Determination of aspirin in pharmaceutical preparations**

test statistic, *t*exp, is

are given in **Table 4** [37].

*t*

with *n*−1 degrees of freedom. The √

Only two laboratories (2 and 9) are within the 95% circle. Because the F-ratio (25.834) is larger than F(0.05,9,9), which is 3.179, it is reasoned that the systematic errors between the analysts are significant at the 95% confidence level.

All this suggests that the variability in the difference between the two samples is evident. There are two factors which influence the variability of the differences. These are the random error of measurement and the heterogeneity of the materials. It is easy to infer that if the materials were relatively heterogeneous, then the reported differences would show considerable variability. That this is not so in this example is evidence that the laboratories were dealing with relatively homogeneous materials, i.e. the composition of the samples of materials X and Y received by each laboratory was essentially the same.

The larger the random error of measurement associated with an analytical procedure, the more varied the results of replicate measurements are. A relatively large random error of measurement will also cause the differences between them to vary considerably. The example presented herein indicates that the observed differences are approximately the same. This suggests that the random error experienced in each laboratory is relatively small. Perpendicular dispersions to the bisector, for homogeneous materials, are reflections of the within-laboratory variability. It is worth mentioning at this point that the example introduced here reveals that the withinlaboratory variabilities are small, compared with the systematic between-laboratory variabilities.

Finally, it is sometimes believed that the centroid in a Youden diagram gives a good estimate of the true values of the two materials. It is the authors' view that this may often not be the case. The scatter of the results obtained by several laboratories around the mean value is a result of both random within-laboratory and systematic between-laboratory variability. Averaging may result in the mean value being close to the true value. It may also result in a mean value which is, for example, much higher than the true value. Suppose, for example, that all the laboratories had a similar positive bias in one of the steps of the procedure. The result will be a scatter about a mean value which maybe higher than the true value.

#### **3.4. Determination of methylparaben in surface water by liquid chromatography-negative electrospray ionization tandem mass spectrometry**

Since parabens were discovered due to their antimicrobial activity, they have been widely used as bactericides, fungicides and preservative agents in many cosmetics, pharmaceuticals, personal care products and food, amongst other consumer products.

Although the toxicity of these compounds is very low, they present a weak estrogenic activity and are considered as endocrine disruptors. That is why they have been classified as emerging contaminants attracting scientific attention on a global scale. These compounds, after consumption, reach wastewater treatment plants, where they are not efficiently removed; thus, they end up in the environment. These chemical compounds consist of detergents, soaps and/or other products.

An analytical method for the determination of methylparaben (MeP) in surface water samples is applied during a period of 15 days.

The method is based on solid-phase extraction (SPE) and subsequent analysis by highperformance liquid chromatography-triple quadrupole mass spectrometry (HPLC-QqQ-MS) [38]. The Youden plot has been applied to the results. A detailed description of the completed experimental procedure is shown below:

#### *3.4.1. Experimental part*

#### *3.4.1.1. Materials and reagents*

HPLC-grade water, acetone and methanol were purchased by a company in Spain. Analyticalgrade formic acid (98%), sulphuric acid (97%) and hydrochloric acid (37%) were acquired from another specialist industry in Spain. Ammonium acetate, MeP (≥99%), was bought from a firm in the USA.

Three millilitres SPE cartridges, packed with 60 mg of Oasis HLB, were purchased from Waters (Milford, MA, USA).

Stock solution, at a concentration of 1000 mg L−1, was prepared in methanol and stored at 4°C. Working solutions were prepared by diluting the stock standard solutions in methanol.

#### *3.4.1.2. Sample collection*

Surface water samples were collected in May 2016 and were taken from Guadalquivir River (Seville, Spain). These samples were collected in amber glass bottles precleaned with acetone and methanol. In order to stabilize them, acetonitrile was immediately added after sampling to achieve a final concentration of 0.5% v/v. Stabilized samples were stored at 4°C until further analysis, which was carried out within 48 h after sample collection. Prior to extraction, samples were filtered through a 1.2 μm glass-fibre membrane filter supplied by a British manufacturer.

#### *3.4.1.3. Solid-phase extraction*

be the case. The scatter of the results obtained by several laboratories around the mean value is a result of both random within-laboratory and systematic between-laboratory variability. Averaging may result in the mean value being close to the true value. It may also result in a mean value which is, for example, much higher than the true value. Suppose, for example, that all the laboratories had a similar positive bias in one of the steps of the procedure. The

**3.4. Determination of methylparaben in surface water by liquid chromatography-negative** 

Since parabens were discovered due to their antimicrobial activity, they have been widely used as bactericides, fungicides and preservative agents in many cosmetics, pharmaceuticals,

Although the toxicity of these compounds is very low, they present a weak estrogenic activity and are considered as endocrine disruptors. That is why they have been classified as emerging contaminants attracting scientific attention on a global scale. These compounds, after consumption, reach wastewater treatment plants, where they are not efficiently removed; thus, they end up in the environment. These chemical compounds consist of detergents, soaps and/or other

An analytical method for the determination of methylparaben (MeP) in surface water samples

The method is based on solid-phase extraction (SPE) and subsequent analysis by highperformance liquid chromatography-triple quadrupole mass spectrometry (HPLC-QqQ-MS) [38]. The Youden plot has been applied to the results. A detailed description of the completed

HPLC-grade water, acetone and methanol were purchased by a company in Spain. Analyticalgrade formic acid (98%), sulphuric acid (97%) and hydrochloric acid (37%) were acquired from another specialist industry in Spain. Ammonium acetate, MeP (≥99%), was bought from

Three millilitres SPE cartridges, packed with 60 mg of Oasis HLB, were purchased from

Stock solution, at a concentration of 1000 mg L−1, was prepared in methanol and stored at 4°C. Working solutions were prepared by diluting the stock standard solutions in methanol.

Surface water samples were collected in May 2016 and were taken from Guadalquivir River (Seville, Spain). These samples were collected in amber glass bottles precleaned with acetone and methanol. In order to stabilize them, acetonitrile was immediately added after sampling

result will be a scatter about a mean value which maybe higher than the true value.

**electrospray ionization tandem mass spectrometry**

66 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

is applied during a period of 15 days.

experimental procedure is shown below:

*3.4.1. Experimental part*

a firm in the USA.

*3.4.1.1. Materials and reagents*

Waters (Milford, MA, USA).

*3.4.1.2. Sample collection*

products.

personal care products and food, amongst other consumer products.

Oasis HLB cartridges were conditioned using 3 mL of methanol followed by 3 mL of 0.5N hydrochloric acid and 3 mL of de-ionized water. Prior to extraction, the pH of the sample was adjusted to 2 by the addition of sulphuric acid 40% (v/v). The acidified sample (250 mL) was percolated through the cartridge at a flow rate of approximately 10 mL/min−1. Then, the volumetric flask containing the sample was rinsed with 5 mL of de-ionized water, and the extract was added to the cartridge.

After loading the cartridges, they were washed with 5 mL of de-ionized water and dried for 10 min. The elution of the analytes was carried out with four successive aliquots of 1 mL of methanol at a flow rate of about 1 mL/min. The eluates were collected in 10-mL collection tubes and evaporated to dryness at room temperature by a gentle nitrogen stream. Finally, the extracts were reconstituted in 1 mL of methanol, filtered through a 0.45 μm nylon filter, and a 20-μL aliquot was injected into the HPLC instrument.

#### *3.4.1.4. High-performance liquid chromatography-mass spectrometry*

Separation was carried out using an Agilent 1200 series HPLC chromatography system equipped with a vacuum degasser, a binary pump, an autosampler and a thermostated column compartment. MeP was isolated with a Zorbax Eclipse XDB-C18 Rapid Resolution HT (4.6 mm × 50 mm i.d.; 1.8-μm particle size) column, using an isocratic elution with methanol (30%) and aqueous 5 mM ammonium acetate solution (70%) as mobile phase. Flow rate was 0.6 mL/min. The injection volume was 20 μL. The column temperature was maintained at 25°C.

The HPLC system was coupled to a 6410 triple quadrupole (QqQ) mass spectrometer (MS) equipped with an electrospray ionization source operating in negative mode. Two transitions were used for its identification (92.1m/z) and confirmation (136.1m/z).

The ionization of analytes was carried out using the following settings:


#### *3.4.2. Results and discussion*

The results for the different days are shown in **Table 5**.

Because the F-ratio (3.000) is larger than F(0.05,14,14), which is 2.484, it is concluded that the systematic errors between the analysts are significant at the 95% confidence level.

**Figure 7** depicts the results. By observing it, one may see that most of points fall in either the first or the third quadrant. Two days (5 and 11) are outside the 95% circle. The following comments may be drawn hereto:


Finally, the z-score approximation is applied (pls. refer to Section 2.2) [39]. The results are shown in **Table 6**. A comparison of the results obtained by the laboratories using the Youden plot and Z-Score is done as follows:


**Table 5.** Determination of MeP in two similar surface water samples in different days.

**Figure 7.** Youden plot (determination of MeP).

*3.4.2. Results and discussion*

ments may be drawn hereto:

plot and Z-Score is done as follows:

The results for the different days are shown in **Table 5**.

68 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

although the latter is the very edge of the circle.

– Day 6 is within the circle but at the very edge of it.

Because the F-ratio (3.000) is larger than F(0.05,14,14), which is 2.484, it is concluded that the

**Figure 7** depicts the results. By observing it, one may see that most of points fall in either the first or the third quadrant. Two days (5 and 11) are outside the 95% circle. The following com-

– Days 2, 3 , 8, 10 and 13 are within the 95% circle, close together, in the third quadrant,

Finally, the z-score approximation is applied (pls. refer to Section 2.2) [39]. The results are shown in **Table 6**. A comparison of the results obtained by the laboratories using the Youden

**X Y X Y**

systematic errors between the analysts are significant at the 95% confidence level.

– Days 4, 14 and 15 are within the 95% circle, near each other in the first quadrant.

– Days 1, 9 and 12 are within the 95% circle, close together but in different quadrants.

– Day 7 is also within the circle but farther away from the other days.

**Day Area MeP (HPLC-MS/MS) MeP concentration (ng/L)**

 3329 3574 16.1 17.8 3255 3388 15.6 16.5 3224 3302 15.3 15.9 3518 3886 17.4 20.1 3621 4095 18.2 21.6 3738 3435 19.0 16.8 3108 3862 14.5 19.9 3145 3200 14.8 15.2 3205 3531 15.2 17.5 3266 3133 15.6 14.7 3056 3076 14.1 14.3 3468 3537 17.1 17.6 3065 3162 14.2 14.9 3357 3758 16.3 19.1 3417 3877 16.7 20.0

**Table 5.** Determination of MeP in two similar surface water samples in different days.


#### **3.5. Antibody concentrations: an ISO-13528:2005(E) example**

Finally, a confidence ellipse, calculated as described in ISO 13528 [40], has been used as an aid to interpret the plot so as to deal with those situations, in which the two samples differ in magnitude of the property measured. A Youden Plot for the original data may be derived from the z-scores (as explained below). It is constructed by plotting the z-scores obtained on one of the materials against the z-scores obtained on the other material.


**Table 6.** Application of z-score to the determination of MeP in two similar surface water samples.

For ease of reference, let A and B denote the two materials:


 *ZA*,*<sup>i</sup>* <sup>=</sup> (*XA*,*<sup>i</sup>* <sup>−</sup> ¯ *XA* ) \_\_\_\_\_\_\_ *SA* ;  *and ZB*,*<sup>i</sup>* <sup>=</sup> (*XB*,*<sup>i</sup>* <sup>−</sup> ¯ *XB* ) \_\_\_\_\_\_\_ *SB* (18)

3. Calculate the combined scores for the two materials:

\*\*3.\*\* Calculate the combined scores for the two materials:

$$Z\_{A, \mathbb{A}, l} = \sqrt{Z\_{A,l}^2 - 2\oint Z\_{A,l} Z\_{\mathbb{A},l} + Z\_{\mathbb{A},l}^2} \tag{19}$$

4. In terms of the standardized variables, the confidence ellipse may be written in terms of Hotelling's T<sup>2</sup> :

$$Z\_A^2 - 2\oint Z\_A Z\_\mathbf{z} + Z\_\mathbf{z}^2 = \left(1 - \bigwedge^2\right) T^2\tag{20}$$

where

$$T^2 = 2\left\{\frac{(p-1)}{(p-2)}\right\} F\_{(1\text{-}a)}\left(2, p-1\right) \tag{21}$$

Here, *F*(1 − *α*) (2, *p* − 1) is the tabulated (1-α)-fractile of the F-distribution with 2 and (p-1) degrees of freedom.

As recommended by the International Organization for Standardization (ISO), the ellipse may be drawn on a graph with the z-scores ZA and ZB as the axes by plotting a series of points for −T < ZA < T with

for  $-1 \le Z\_A \le 1$  with 
$$Z\_\text{B} = \bigwedge \mathcal{Z}\_A \pm \sqrt{\left(1 - \bigwedge^2 \right) \left(T^2 - Z\_A^2\right)}\tag{22}$$

To interpret the Youden Plot, the combined z-scores may be used. The highest combined z-score corresponds to the highest significance level of 100%.

Also, the combined z-scores aid to identify the outlying points.

For ease of reference, let A and B denote the two materials:

**Day X Y <sup>ε</sup><sup>T</sup> CS CR <sup>ε</sup><sup>S</sup> <sup>ε</sup><sup>R</sup> ET <sup>ε</sup>**

70 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

1 16.1 17.8 0.54 0.53 0.11 0.45 0.09 0.54 0.01 0.52 Satisfactory 2 15.6 16.5 1.03 −0.78 0.67 −0.55 0.48 1.03 0.23 0.99 Satisfactory 3 15.3 15.9 1.67 −1.37 0.95 −0.99 0.68 1.67 0.46 1.60 Satisfactory 4 17.4 20.1 3.11 3.07 0.52 2.66 0.45 3.11 0.2 2.98 Questionable 5 18.2 21.6 4.77 4.65 1.06 3.88 0.89 4.77 0.78 4.57 Unsatisfactory 6 19.0 16.8 3.45 1.9 2.88 1.37 2.08 3.45 4.31 3.31 Unsatisfactory 7 14.5 19.9 2.62 0.87 2.47 0.68 1.94 2.62 3.76 2.52 Questionable 8 14.8 15.2 2.52 −2.29 1.07 −1.72 0.80 2.52 0.64 2.42 Questionable 9 15.2 17.5 0.44 −0.31 0.31 −0.22 0.22 0.44 0.05 0.42 Satisfactory 10 15.6 14.7 2.85 −2.02 2.02 −1.43 1.43 2.85 2.03 2.73 Questionable 11 14.1 14.3 3.59 −3.36 1.24 −2.62 0.97 3.59 0.93 3.44 Unsatisfactory 12 17.1 17.6 1.44 1.05 0.99 0.74 0.7 1.44 0.49 1.38 Satisfactory 13 14.2 14.9 3.01 −2.88 0.85 −2.32 0.69 3.01 0.47 2.88 Questionable 14 16.3 19.1 1.75 1.61 0.68 1.23 0.52 1.75 0.27 1.68 Satisfactory 15 16.7 20.0 2.`70 2.51 0.98 1.94 0.76 2.70 0.58 2.59 Questionable

2. Calculate the z-scores for the two materials as follows:

3. Calculate the combined scores for the two materials:

2

coefficient ( *ρ*^ ).

Hotelling's T<sup>2</sup>

where

*ZA*,*<sup>i</sup>* <sup>=</sup> (*XA*,*<sup>i</sup>* <sup>−</sup>

*ZA*,*B*,*<sup>i</sup>* = √

*T*<sup>2</sup> = 2 {

:

*ZA*

1. Calculate the averages and standard deviations of the two sets of data and the correlation

;  *and ZB*,*<sup>i</sup>* <sup>=</sup> (*XB*,*<sup>i</sup>* <sup>−</sup>

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

,*<sup>i</sup>* − 2 *ρ*^ *ZA*,*<sup>i</sup> ZB*,*<sup>i</sup>* + *ZB*

<sup>2</sup> = (1 − *ρ*^

¯ *XB* ) \_\_\_\_\_\_\_ *SB*

**R 2**

 **Z-score**

2 , *i*

2

(18)

(19)

) *T*<sup>2</sup> (20)

(2, *p* − 1) (21)

¯ *XA* ) \_\_\_\_\_\_\_ *SA*

**Table 6.** Application of z-score to the determination of MeP in two similar surface water samples.

*ZA* 2 

− 2 *ρ*^ *ZA ZB* + *ZB*

(*p* - 1) \_\_\_\_ (*p* - 2) } *F*(1-*α*)

4. In terms of the standardized variables, the confidence ellipse may be written in terms of

When a Youden Plot is constructed, it may be interpreted as follows:


**Table 7** shows data obtained by testing two similar samples for antibody concentrations and the calculations required to derive the confidence ellipse. With p = 29 laboratories and using



Units (U) in thousands (k) per litre (l) of sample, where a unit is defined by the concentration of an international reference material


**Table 7.** Data and calculations on concentrations of antibodies for two similar allergens.

a significance level of 100% = 5%, *F*(1 − *α*) (2, *p* − 1) = 3.34. Hence, T = 2.632. The ellipse is shown, together with the points representing the z-scores, in **Figure 8**, in tandem with the ellipses pertaining to probability levels of 100% = 1% and 0.1%.

As depicted in **Figure 8**, laboratories 5 and 23, with combined z-scores of 1.641 and 2.099, respectively, are found in the top right-hand quadrant. Laboratory 26 has a high z-score on material B (2.019) compared to material A (−0.055) and a combined z-score of 2.059 followed by laboratory 8 with a combined z-score of 1.501. The points for laboratories 23 and 26 fall between the ellipses for the 5% and 1% probability levels. Thus, the results may be perceived as giving rise to warning signals.

**Figure 8.** Youden plot of z-scores from **Table 7** (concentrations of antibodies for two similar allergens).

## **4. Conclusions**

a significance level of 100% = 5%, *F*(1 − *α*)

reference material Hotelling's *T*<sup>2</sup> 6.927 *T* 2.632 *ρ*̂ 0.706

*F* (5%) 3.34

as giving rise to warning signals.

pertaining to probability levels of 100% = 1% and 0.1%.

Mean 11.543 7.659 0.000 0.000 Standard deviation 3.294 2.897 1.000 1.000

**Table 7.** Data and calculations on concentrations of antibodies for two similar allergens.

(2, *p* − 1) = 3.34. Hence, T = 2.632. The ellipse is shown,

together with the points representing the z-scores, in **Figure 8**, in tandem with the ellipses

Units (U) in thousands (k) per litre (l) of sample, where a unit is defined by the concentration of an international

**Row Data Z-score Combined Z-score Allergen A (U) Allergen B (U) ZA ZB ZAB**

 11.730 5.770 0.057 −0.652 0.693 12.290 6.970 0.227 −0.238 0.429 10.950 6.230 −0.180 −0.493 0.388 10.950 5.900 −0.180 −0.607 0.497 11.170 7.740 −0.113 0.028 0.134 11.200 8.630 −0.104 0.335 0.415 7.640 3.740 −1.185 −1.353 0.985 12.170 7.330 0.190 −0.114 0.282 10.710 5.700 −0.253 −0.676 0.529 7.840 6.070 −1.124 −0.549 0.833 20.470 15.660 2.710 2.762 2.098 12.600 11.760 0.321 1.415 1.210 11.370 4.910 −0.052 −0.949 0.913 11.360 13.510 −0.055 2.019 2.059 10.750 5.480 −0.241 −0.752 0.607 12.210 9.770 0.203 0.729 0.603 7.490 5.820 −1.230 −0.635 0.902

72 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

As depicted in **Figure 8**, laboratories 5 and 23, with combined z-scores of 1.641 and 2.099, respectively, are found in the top right-hand quadrant. Laboratory 26 has a high z-score on material B (2.019) compared to material A (−0.055) and a combined z-score of 2.059 followed by laboratory 8 with a combined z-score of 1.501. The points for laboratories 23 and 26 fall between the ellipses for the 5% and 1% probability levels. Thus, the results may be perceived Intercomparison exercises are of great value in systems of quality assessment allowing the examination of the analytical process and the generated results. Youden plot or Youden analysis is particularly aimed at interlaboratory comparisons, obtaining accurate information without much effort. The main characteristic is its ability to separate the systematic and random errors with minimal effort on the part of the participants. To implement the method:


 polyunsaturated fatty acids in fats and oils; total blood cholesterol; and aspirin in pharmaceutical preparations, respectively.


## **Abbreviation list**


## **Author details**

Julia Martín, Nieves Velázquez and Agustin G. Asuero\*

\*Address all correspondence to: asuero@us.es

Department of Analytical Chemistry, Faculty of Pharmacy, The University of Seville, Seville, Spain

## **References**

[1] Funk W, Dammann V, Donnevert G. Quality assurance in analytical chemistry application in environmental, food and materials analysis, biotechnology and medical engineering. 2nd ed., Wiley-VCH: Weinheim, Germany, 2007; pp. 179–188.

[2] Horwitz W. Protocol for the design, conduct and interpretation of collaborative studies. Pure Appl. Chem. 1995; **67**(2):331–343.

polyunsaturated fatty acids in fats and oils; total blood cholesterol; and aspirin in pharma-

• In most cases, it is observed that systematic errors may be regarded as the main cause of variation with most of points in quadrants (+ +) and (− −). Finally, a detailed procedure for the determination of methylparaben in surface waters, of special relevance nowadays in the environmental field, has been developed by liquid chromatographytandem mass spectrometry. In this experimental system, an alternative to the Youden method based on the z-score has also been assessed showing no discrepancy between

• A confidence ellipse is proposed in ISO 13528:2005 to deal with those situations where the two samples differ in magnitude of the property measured. The extension of the Youden method based on the confidence ellipse may be used as the building platform for further

Department of Analytical Chemistry, Faculty of Pharmacy, The University of Seville, Seville,

[1] Funk W, Dammann V, Donnevert G. Quality assurance in analytical chemistry application in environmental, food and materials analysis, biotechnology and medical engi-

neering. 2nd ed., Wiley-VCH: Weinheim, Germany, 2007; pp. 179–188.

studies, incorporating amongst other three- and four-dimensional Youden Plots.

ceutical preparations, respectively.

74 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

BF Boron trifluoride-methanol

PUFA Polyunsaturated fatty acids

Julia Martín, Nieves Velázquez and Agustin G. Asuero\*

\*Address all correspondence to: asuero@us.es

MeP Methylparaben

MS Mass spectrometer

QqQ Triple quadrupole

SPE Solid-phase extraction

FDA Food and Drug Administration

ISO International Organization for Standardization

both methods.

**Abbreviation list**

**Author details**

Spain

**References**


[32] Kunsagi Z, Stroka J. Report on the validation of a method for the determination of Ochratoxin A in Capsicum spp. (Paprika and chilli). European Commission, Joint Research Centre, Institute for Reference Materials and Measurements: Geel, Belgium, 2012.

[18] Youden WJ, Steiner EH. Statistical manual of the association of official analytical chemists. Statistical techniques for collaborative tests. Planning and analysis of results of col-

[19] Juran JM, Godfrey AB. (Eds.) Juran's quality handbook, 5th ed.. McGraw-Hill: New

[20] Zhong Z, Li G, Luo J, Chen W, Liu L, He P, Luo Z. Proficiency testing for determination of lead and arsenic in cosmetics: comparison of analytical procedures and evaluation of

[21] Juneja A, Anand A.Understanding statistical concepts in laboratory quality control mea-

[22] Youden WJ. Experimentation and measurement. National Institute of Standards and Technology, NIST, Special Publication 672. U.S. Department of Commerce. Reprinted;

[23] Youden WJ. Graphical diagnosis of interlaboratory test results. Ind. Qual. Control. 1959;

[24] Youden WJ. Statistical aspects of the cement testing program. Proc. Am. Soc. Testing

[25] Youden WJ. Evaluation of chemical analyses on two rocks. Technometrics. 1959;

[26] Tracy ND, Young JC. A bivariate control chart for paired measurements. J. Qual. Technol..

[27] Mandel J, Lashof TW. Interpretation and generalization of Youden's two- sample.

[28] Cornell JA. The man and his methodology. AQQC Statistical Division Newsletter. 1993;

[29] Lavetz R. Statistical Manual. Chemical Proficiency Testing – NMI north ryde. Chemical & biological metrology. Document No 3. Australian Government. National Measurement

[30] Starink RJ and Visser RG. Interlaboratory studies: protocol for the organisation, statistics and evaluation. Institute for Interlaboratory Studies. Spijkenisse, The Netherlands; 2014.

[31] De Girolamo A, Ciasca B, Stroka J, Bratinova S, Visconti A, La anzio VMT. Determination of DON, FB1, FB2, ZEA, T-2, HT-2, OTA, AFB1, AFG1, AFB2, AFG2 in maize and determination of DON, ZEA, T-2, HT-2, OTA in wheat. Report of the 2014 Proficiency Test for LC-MS(MS) multi-mycotoxin methods. National Research Council of Italy. Institute of

sures in biomedical research. Int. J. Res. Med. Sci.. 2015; **3**(11):3443–3445.

laborative tests. AOAC: Washington DC, 1975.

76 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Mat. Philadelphia, PA.. 1959; **59**: 1120–1128.

1977.

**15**(11):24–28.

**1**(4):409–417.

**13**(2):9–18.

pp. 1–27.

1995; **27**(4):370–376.

Institute; 2014. pp. 1–15.

J. Qual. Technol.. 1974; **6**(1):22–36.

Sciences of Food Production. Italy; 2014.

York, 1999; Youden Two-Sample Plan, 47.28.–47.29.

laboratory performances. Anal. Methods. 2015; **7**(7):3196–3177.


D. Second interlaboratory exercise on non- steroidal anti-inflammatory drug analysis in environmental aqueous samples. Talanta. 2010; **81**:1189–1196.

[58] Pankratov I, Elhanany S, Hening S, Zaritsky S, Ostapenlo I, Kuselman I. Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis. Accred. Qual. Assur.. 2010; **15**(8):459–466.

[45] Van der Bresselaar AMHP, Abdoel CF, Ardanary D, van de Kamp G, Versluijs FAC. Preparation and control blood for external quality assessment of point-of-care international normalized ratio testing in the Netherland. Am. J. Clin. Pathol.. 2014; **14**:879–883.

[46] Xiao YL, Zhang CB, Zhao HJ, Kang FF, Wang W, Zhong K, Yuan S, Wang ZG. Application of ISO 13528 robust statistical methods for external quality assessment of blood glucose

[47] O'Donnel GE, Hibbert DB. A study of the conditions of measurement required to evaluate bias in analytical results illustrated by the use of data from a multi-round, blind-

[48] Monteiro LR, Grafitti D, Albano F, Porfírio D, Fernandes Jr LP, Cotrim MEB, Pires MAF. Evaluation of a Brazilian ion chromatography interlaboratory study.Accred. Qual.

[49] Shirono K, Iwase K, Okazaki H, Yamazawa M, Shikakume K, Fukumoto N, Murakami M, Yanagisawa M, Tsugoshi T. A study on the utilization of the Youden plot to evaluate

[50] Yu W, Tan Q, Yu L, Cao H. Comparison of statistical model of Youden plot for profi-

[51] Bremser W, Lücke FK, Urmetzer C, Fuchs E, Leist U. An approach to integrated data assessment in a proficiency test on the enumeration of *Escherichia coli*. J. Appl. Microbiol..

[52] de la Calle Guntiñas MB, Semeraro A, Wysocka I, Cordeiro F, Quétel C, Emteborg H, Charoud-Got J, Linsinger TPJ. Proficiency test for the determination of heavy metals in mineral feed. The importance of correctly selecting the certified reference materials during method validation. Food Addit. Cont. Part A Chem. Anal. Control Expo. Risk

[53] Kanefuji K, Tsugoshi T, Iwase K. Evaluation between statistical methods relate to the z

[54] Lavine BK. Learning Outcome Assessments in Quantitative Analysis Lab Using Youden Plots. New Trends in the Teaching of Analytical Chemistry. FACSS Analytical Science and Innovation. 2011 . https://www.scixconference.org/program/archive?p=4424&yearS

[55] Mann I, Brookman B. Selection, use and interpretation of proficiency testing (PT)

[56] Wang JS, Kee MK, Choi BS, Kim CW, Kim SS. Evaluation of external quality assessment results for HIV testing laboratories in Korea using current analytical methods. Clin.

[57] Heath E, Kosjek T, Farre M, Quintana JB, De Alecastro LF, Castiglioni S, Gans O, Langford K, Loos R, Radjenovic J, Rocca LM, Budzinski H, Tsipi D, Petrovic M, Barcelo

score for use in proficiency testing. Bunseki Kagaku. 2011; **60**(7):571–577.

measurements in China. Accred. Qual. Assur.. 2014; **19**(5):397–401.

proficiency test results. Accred. Qual. Assur.. 2013; **18**(3):161–174.

ciency testing in ISO13528 and NATA. Metall. Anal.. 2013; **33**(12):74–80.

duplicated, proficiency test. Analyst. 2013; **138**:3673–3678.

Assur.. 2013; **18**(3):207–215.

78 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

2011; **110**(1):128–138.

elect=2011

Assess.. 2011; **28**(11):1534–1546.

Schemes. 2nd ed. EA-Eurolab-Eurachem; 2011.

Chim. Acta. 2011; **412**(11–12):1127–1132.


[84] Gadmar TC, Vogt RD, Osterhus B. The merits of the high-temperature combustion method for determining the amount of natural organic carbon in surface freshwater samples. Inter. J. Environ. Anal. Chem. 2002; **82**(7):451–461.

[70] Lau A. Practical advice on Youden plot. Qual. Progress. 2007; **40**(10):10.

mass spectrometry. Clin. Chim. Acta. 2006; **366**:249–256.

tory comparison. Accred. Qual. Assur. 2006; **11**(8):462–469.

Portland cement. Accred. Qual. Assur. 2006; **11**(8):414–421.

elements in serum. Ann. Ist. Super. Sanita. 2005; **41**(1):171–179.

**90**(1):6–20.

**28**(3–4):537–544.

2005; **65**(4):1022–1030.

Assur.. 2004; **9**(1):5–17.

ment process. Clin. Chem.. 2003; **49**(3):463–469.

SP-531), 9–13 December, Frascati, Lacoste, H. (Ed.), Italy, 2003.

**88**(6):1840–1841.

pp. 2–12.

Biochem.. 1982; **20**(12):883–892.

80 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

[71] Miller EL, Bimbo AP, Barlow SM, Sheridau B. Repeatability and reproducibility of determination of the nitrogen content of fishmeal by the combustion (Dumas) method and comparison with the Kjeldahl method: interlaboratory study. J. AOAC Int.. 2007;

[72] Siekmann L, Breuer H. Determination of cortisol in human plasma by isotope dilutionmass spectrometry. Definitive methods in clinical chemistry, I. J. Clin. Chem. Clin.

[73] Sorensen K, Grung M, Röttgers R. An intercomparison of in vitro chlorophyll A determinations for MERIS level 2 data validation. Int. J. Remote Sensing-MERIS. 2007;

[74] Bons, JA, de Boer, D, van Dieijen-Visser, MP and Wodzig, WK. Standardization of calibration and quality control surface-enhanced laser desorption/ionization time of flight

[75] Schilling P, Powilleit M, Uhlig S. Chlorophyll-a determination: results of an interlabora-

[76] Svegl F, Strupi JS, Svegl IG. Proficiency testing of chloride content in different types of

[77] Costantini S, Ciaralli L, Ciprotti M, D'Ilio S, Giordano R, Mosca M, Sepe A, Jenofonte O. The network of the Italian laboratories: a proficiency test on the quantification of trace

[78] González AG, Herrador MA, Asuero AG. Practical digest for evaluating the uncertainty of analytical assays from validation data according to the LGC/VAM protocol. Talanta.

[79] Jones FE. Youden analysis of Karl Fisher titration data from an interlaboratory study determining water in animal feed, grain and forage. J. AOAC Int.. 2005;

[80] Siekmann L. Establishing measurement traceability in clinical chemistry. Accred. Qual.

[81] Margolis SA, Vangel M, Duewer DL. Certification of standard reference material 970, ascorbic acid in serum, and analysis of associated interlaboratory bias in the measure-

[82] Sorensen K, Grung M, Röttgers R. An intercomparison of in vitro chlorophyll A determination – preliminary results, Proceedings of the ENVISAT Validation Workshop (ESA

[83] Official Methods of Analysis (AOAC). Guidelines for collaborative study procedures to validate characteristics of a method of analysis; 2002 AOAC International; Appendix D,


[113] Mcdonald RW and Nelson H. A laboratory performance check for the determination of metals (Hg, Zn, Cd, Cu,Pb) in reference marine sediments. Canadian Technical Report of Hydrography and Ocean Sciences, No. 33. Institute of Ocean Sciences Department of Fisheries and Oceans, Sidney; 1984.

[98] Horwitz W. Nomenclature of interlaboratory analytical studies. Pure Appl. Chem.

[99] ISO 5725-2:1994. Accuracy (trueness and precision) of measurement methods and results – Part 2: basic method for the determination of repeatability and reproducibility

[100] Stephens RD, Rappe C, Hayward DG, Nygren M, Startin J, EsbØll A, Carlé J, Yrjänheikki EJ. World health organization international intercalibration study on dioxins and

[101] Mesley RJ, Pocklington WD, Walker RF. Analytical quality assurance. A review.

[102] Jones NE. Multiway analysis of variance for the interpretation of interlaboratory stud-

[103] Oosting E, Neugebauer E, Keyzer JJ, Lorenz W. Determination of histamine in human plasma: the European external quality control study 1988. Clin. Exp. Allergy. 1990;

[104] Youden WJ. Classic paper. The collaborative test. (Reprinted from J-Assoc-Off-Agric-Chem, Vol. 46, PG 55-62, 1963). J. Assoc. Off. Anal. Chem.. 1990; **73**(2):194–201.

[105] Thompson M. Robust statistic and functional relationship estimation for comparing the bias of analytical procedures over extended concentration ranges. Anal. Chem.. 1989;

[106] Ferrus R, Torrades F. Bias-free adjustment of analytical methods to laboratory samples

[107] Abern AM, Garrell RL. Exchange of comments on a new technique in chemical assay

[108] Bauer CF, Grant CL, Jenkins TF. Interlaboratory evaluation of high-performance liquid chromatographic. Determination of nitroorganics in munition plant wastewater. Anal.

[109] Boyer KW, Horwitz W, Albert R. Interlaboratory variability in trace element analysis.

[110] Currie LA. The limitations of models and measurements as revealed through chemo-

[111] Cleveland WS and McGrill R.The many faces of a scatterplot. J. Am. Stat. Assoc.. 1984;

[112] Jenkins TF, Leggett DC, Grant CL, Bauer CF. Reversed-phase high-performance liquid chromatographic determination of nitroorganics in munitions wastewater. Anal.

metric intercomparison. J. Res. Natl. Bur. Stand. 1985; **90**(6):409–419.

in routine analytical procedures. Anal. Chem.. 1988; **69**(13):1281–1285.

calculations. Anal. Chem.. 1987; **59**(23):2816–2818.

of a standard measurement method. 1994. https://www.iso.org/

furans in human milk and blood. Anal. Chem.. 1992; **64**(24):3109–3117.

1994; **66**(9):1903–1911.

82 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Analyst. 1991; **116**(10):975–990.

**20**:349–357.

**61**(17):1942–1945.

Chem.. 1986; **58**(1):176–182.

**79**(388):807–822.

Chem.. 1986;**58**:170–175.

Anal. Chem.. 1985; **57**(2):454–459.

ies. Anal. Chem.. 1990; **62**:1532–1536.


## **Using Lot Quality Assurance Sampling to Monitor the Prevalence of Abortions and the Quality of Reproductive Health Care in Armenia Using Lot Quality Assurance Sampling to Monitor the Prevalence of Abortions and the Quality of Reproductive Health Care in Armenia**

Joseph J. Valadez and Lusine Mirzoyan Joseph J. Valadez and Lusine Mirzoyan

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66092

#### **Abstract**

Monitoring abortion prevalence is essential to plan control efforts. Lot Quality Assurance Sampling (LQAS) is an inexpensive, reliable method for monitoring abortion prevalence and access to quality reproductive health (RH) services. This chapter presents survey results from 2000 in three sites of Armenia (Gyumri, Gavar and Goris) using LQAS principles (i.e., 44%, 95% CI: ±6% of women had an induced abortion in their lifetime, a total abortion rate (TAR) of 2.0 abortions per woman). Modern contraceptive use was lowest in Goris (16%. 95% CI: ±7%) and highest in Gyumri (43%, 95% CI: ±11%). Only 37% (95% CI: ±9%) of women with an induced abortion received family planning information and 21% (95% CI: ±4%) of mothers were counselled about family planning after delivery. While limited access to family planning information and contraceptives is still an issue in Armenia, recently new reproductive health priorities—such as infertility, sex-selective abortions and abortions due to socio-economic difficulties—have become more common and can be investigated using LQAS in both community surveys and health facility assessments. This study demonstrates that measuring national abortion prevalence and access to services mask underlying variations; the awareness of which is essential for health program planning.

**Keywords:** abortions, reproductive health, Armenia, lot quality assurance sampling (LQAS), surveys

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## **1. Introduction**

According to the World Health Organization (WHO), an estimated 225 million women in developing countries wish to postpone or prevent bearing a child, but do not use any modern means of contraception [1]. Having become pregnant, many women terminate their pregnancy. In developing countries, 56% of all abortions are unsafe, compared with just 6% in the developed world [2]. Although legal and illegal medication abortions have become more common and have likely contributed to reductions of severe morbidity and maternal death, unsafe abortions in low- and middle-income countries still remain among the leading causes of maternal morbidity and mortality. The ethical aspects of terminating a pregnancy for nonmedical reasons [3–5] are surely another concern.

Abortions, especially when repeated, pose many health risks to a woman, including the risk of death. Women who choose an abortion are approximately four times more likely to die in the following year than women who carry their pregnancies to term [6]. Studies of records for an entire female population in Denmark showed that the risk of death remains higher in each of the first ten years following the abortion. Moreover, the risk of death increases with each abortion: 45% after one abortion, 114% after two abortions and 192% after three or more abortions [7]. The risk of cervical cancer is 2.3 times higher in women with a history of one abortion and 4.92 times higher in women with a history of two or more abortions as compared to women with no history of abortion. Other studies have found a similar trend for the risks of subsequent ovarian and liver cancers; the elevated risk exists for both single and multiple abortions [8–11]. Approximately, 10% of women who abort will suffer immediate complications, of which approximately one-fifth are considered life-threatening [12]. The most common complications after abortions in later pregnancies include placenta previa, pre-term deliveries, handicapped newborns and ectopic pregnancies [13].

Abortions in the former Soviet states traditionally have been the main method of birth spacing, and Armenia, the focus of this chapter, has a similar profile in this regard [14]. According to the Armenia Demographic and Health Survey (ADHS) 2010, despite the decline over 10 years in the proportion of pregnancies resulting in induced abortion (a reduction from 55% in 2000 to 45% in 2005, and to 29% in 2010), 31% of all women of reproductive age reported having at least one induced abortion, with approximately two-thirds having had more than one abortion in their lifetime [15, 16]. The total abortion rate (TARe1 ) for Armenia in 2010 was 0.8 abortions per woman, which is lower than 1.8 in 2005 and significantly lower than 2.6 in 2000; this trend, however, might be attributed to the significant decline in the pregnancy rate over the same period [16]. This report highlights the fact that with the introduction of ultrasound determination of an unborn child's sex, gender-selective abortions became a common procedure in Armenia. In ADHS 2010, 8% of women having an abortion in the three years prior to the survey reported the desire to deliver a boy as the main reason for the termination.

<sup>1</sup> Total abortion rate (TAR) is "the total number of abortions a woman will have in her lifetime if current levels persist. This lifetime risk is a cohort measure and can be calculated with age-specific abortion rates or approximated by multiplying the abortion rate by the length of the reproductive period (30–35 year range) (MEASURE Evaluation: http:// www.cpc.unc.edu/measure/prh/rh\_indicators/specific/pac/abortions-per-1-000-live-births).

UNFP's report concurs by documenting that "with 114 boys born for every 100 girls in 2012, Armenia has one of the highest sex imbalance levels in the world…The imbalance is particularly dramatic for third births: the record level of 173 sons born for every 100 daughters has no known equivalence anywhere else in the world" [17]. Gender-specific abortion is particularly a concern in countries like Armenia having a low total fertility rate (TFRe2 ). Being below replacement levels at 1.7 live births per woman in 2010 the gender-imbalance can result in an increased reduction in the population size [18, 19].

**1. Introduction**

medical reasons [3–5] are surely another concern.

86 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

handicapped newborns and ectopic pregnancies [13].

in their lifetime [15, 16]. The total abortion rate (TARe1

1

According to the World Health Organization (WHO), an estimated 225 million women in developing countries wish to postpone or prevent bearing a child, but do not use any modern means of contraception [1]. Having become pregnant, many women terminate their pregnancy. In developing countries, 56% of all abortions are unsafe, compared with just 6% in the developed world [2]. Although legal and illegal medication abortions have become more common and have likely contributed to reductions of severe morbidity and maternal death, unsafe abortions in low- and middle-income countries still remain among the leading causes of maternal morbidity and mortality. The ethical aspects of terminating a pregnancy for non-

Abortions, especially when repeated, pose many health risks to a woman, including the risk of death. Women who choose an abortion are approximately four times more likely to die in the following year than women who carry their pregnancies to term [6]. Studies of records for an entire female population in Denmark showed that the risk of death remains higher in each of the first ten years following the abortion. Moreover, the risk of death increases with each abortion: 45% after one abortion, 114% after two abortions and 192% after three or more abortions [7]. The risk of cervical cancer is 2.3 times higher in women with a history of one abortion and 4.92 times higher in women with a history of two or more abortions as compared to women with no history of abortion. Other studies have found a similar trend for the risks of subsequent ovarian and liver cancers; the elevated risk exists for both single and multiple abortions [8–11]. Approximately, 10% of women who abort will suffer immediate complications, of which approximately one-fifth are considered life-threatening [12]. The most common complications after abortions in later pregnancies include placenta previa, pre-term deliveries,

Abortions in the former Soviet states traditionally have been the main method of birth spacing, and Armenia, the focus of this chapter, has a similar profile in this regard [14]. According to the Armenia Demographic and Health Survey (ADHS) 2010, despite the decline over 10 years in the proportion of pregnancies resulting in induced abortion (a reduction from 55% in 2000 to 45% in 2005, and to 29% in 2010), 31% of all women of reproductive age reported having at least one induced abortion, with approximately two-thirds having had more than one abortion

per woman, which is lower than 1.8 in 2005 and significantly lower than 2.6 in 2000; this trend, however, might be attributed to the significant decline in the pregnancy rate over the same period [16]. This report highlights the fact that with the introduction of ultrasound determination of an unborn child's sex, gender-selective abortions became a common procedure in Armenia. In ADHS 2010, 8% of women having an abortion in the three years prior to the survey

 Total abortion rate (TAR) is "the total number of abortions a woman will have in her lifetime if current levels persist. This lifetime risk is a cohort measure and can be calculated with age-specific abortion rates or approximated by multiplying the abortion rate by the length of the reproductive period (30–35 year range) (MEASURE Evaluation: http://

reported the desire to deliver a boy as the main reason for the termination.

www.cpc.unc.edu/measure/prh/rh\_indicators/specific/pac/abortions-per-1-000-live-births).

) for Armenia in 2010 was 0.8 abortions

As the TAR has important policy and demographic implications, low- and middle-income countries require an inexpensive and reliable method to monitor TAR, as well as the means to monitor the use of modern family planning methods and the availability and quality of reproductive health (RH) services. However, when measured at a national level, TAR can mask the sub-national variations that exist. Such variations affect policy formation and should be identified to assure the needs of differing subpopulations are addressed [20]. Lot Quality Assurance Sampling (LQAS) is one method that can be used locally for routine monitoring that produces reliable data for managing health programs and for improving health services [21]. With respect to preventing abortions in Armenia, LQAS can serve policy makers and planners by providing much-needed information specific for each region of the country [22, 23].

Although LQAS surveys can produce data of comparable quality to the demographic and health surveys (DHS) [24], they are intended to complement those surveys rather than compete with them. While DHS and multi-country indicator surveys (MICS) are international goldstandard survey methods, they are costly and they measure national indicators at approximately five-year intervals. LQAS is used for a related but different purpose: to frequently track priority indicators related to program coverage, the use of services and/or the quality of care in order to improve program services. Although LQAS data can be aggregated to produce prevalence measures for indicators used at the central level for policy making, its major advantages are to detect the variations in coverage at the sub-national level and to determine if targets and standards of care are met [25, 26].

While LQAS surveys are routinely implemented in many countries of Latin America, Africa and East and Southern Asia to assess a variety of public health programs and the quality of health care, they are less often used in the countries of Eastern Europe, South Caucasus or Central Asia [27, 28]. This chapter reports on a reproductive health survey carried out in three sites in three different regions of Armenia. While this survey was carried out in 2000, it is one of the first documented studies of abortion prevalence in the South Caucasus, and its results are still timely and pertinent to current global health issues.

The study was conducted as a baseline assessment for the USAID-funded Network for Health in Armenia project, which aimed to increase women's access to reproductive health and healthy family information and services through a coordinated effort of three private voluntary

<sup>2</sup> Total fertility rate (TFR) is the number of children the average woman would bear in her lifetime if she experienced the currently observed age-specific fertility rates throughout her reproductive years. The fertility rates refer to the three-year period before the survey (Armenia DHS 2010).

organizations (PVOs), government agencies and non-governmental organizations (NGOs). CARE International in Gyumri, Save the Children in Gavar and the Adventists Development and Relief Agency (ADRA) in Goris directed the three pilot sites (or program catchment areas). The pilot sites formed the nucleus of the regional networks and were linked to the central network in the capital under the leadership of ADRA.

#### **2. Study methodology and results**

#### **2.1. Sampling method**

The survey design used the LQAS method, which is an established analysis technique, originally developed as a classification method for industrial quality control during the 1920s and adapted to health sciences in the mid-1980s [29, 30] to manage units in a health system charged with delivering services, referred to in LQAS as supervision areas (SAs). In each SA, a random sample ('n') of individuals is assessed. A decision rule or cut-off value, designated as 'd', is selected to optimize identification of low performance SAs for a specified indicator. The 'd' depends on the sample size, thresholds for classifying high and low performance and the choice of two misclassification errors: the risk of overestimation when low coverage exists (β error) and the risk of underestimation when high coverage exists (α error). The upper threshold, 'p-Upper or pU', for identifying acceptably performing SAs is often suggested by program stakeholders, and the low threshold, 'p-Lower or pL', is usually 30% lower.

In the surveys used for this chapter, 19 locations were selected in each Network SA. The SA sample size, *n* = 19, guarantees that both misclassification errors do not exceed 0.10, as all corresponding values of 'pL' are 30% below 'pU'. SA having pL< p <Pu are classified according to their proximity to either pL or pU. SA with an intermediate performance has an equal chance of being classified as either high or low. The strength of LQAS is that it is very effective for identifying areas that are among the *worst of the worst* in terms of the indicator of interest. As an example, let us assume that the target for an indicator (such as exclusive breastfeeding) is set at 80%. The upper threshold, 'pU', is 80% for identifying high (or acceptably) performing SAs, while the lower threshold, 'pL', is 50%. The sample size 'n=19' and decision rule 'd=13' are selected to ensure α-errors of ≤10%, and β-errors ≤ 10%—or more formally:


The decision rules and error terms associated with varied sample sizes are found in [23, 24].

In Armenia, our intention was to apply LQAS for recurrent community monitoring by each of the three Network's members and the Network as a whole. The program area in each of the three sites was divided into administratively meaningful units or supervision areas. In each case, the Network member SAs were city areas, towns and surrounding villages. A supervisor —such as a doctor, or a nurse, or a midwife—managed each SA. With LQAS, data collectors randomly select small samples (*n*=19) in each SA, which they use to judge performance. These data, when aggregated for a PVO or for the Network, formed a stratified random sample of the members' catchment area. All catchment areas in aggregate form the Network catchment area.

SAs with intermediate performance are classified as high or low depending on how close they fall to the relevant thresholds. There are three major advantages of using LQAS as compared to other probability sampling methods.

First, in addition to permitting calculation of a conventional average coverage for a program area, the method allows program managers to determine the relative performance of the different SAs that comprise the catchment area. For example, a typical health program area could include several communities with a total population of several thousand people. In baseline surveys, the supervisors determine whether any SA is below average and, therefore, needs special assistance. Based on baseline results, the annual (or semi-annual) performance benchmarks are established for recurrent monitoring. Then, in monitoring, the LQAS is used to determine whether SAs reach these performance benchmarks.

Second, LQAS uses a small sample size for making judgements. In addition to necessitating fewer interviews than for other conventional sampling methods, the smaller sample size leads to a quicker analysis and interpretation by local managers. For most applications, a sample of 19 individuals is required in each SA to judge whether it is below average or has reached a service delivery target. For calculation of a coverage proportion for the catchment area, the individual SA level results produced with the samples of 19 are aggregated while being weighted by the population size of the SA. Assuming there are five SAs, the total sample size would be 95. With *p*=50%, this sample results in a coverage measure for the catchment area having a confidence interval that does not exceed ±10% of the true coverage. Ideally it should be five or more SAs; however, having four SAs is also acceptable since the confidence interval increases only slightly (≤11%).

Third, as LQAS uses a small sample to judge whether a health worker's performance reaches a predetermined standard, data collection in densely populated areas does not seriously compete for time health workers can allocate to other health care activities. Hence, it is a practical management tool that is ideal for recurrent data collection.

#### **2.2. Study procedures**

organizations (PVOs), government agencies and non-governmental organizations (NGOs). CARE International in Gyumri, Save the Children in Gavar and the Adventists Development and Relief Agency (ADRA) in Goris directed the three pilot sites (or program catchment areas). The pilot sites formed the nucleus of the regional networks and were linked to the central

The survey design used the LQAS method, which is an established analysis technique, originally developed as a classification method for industrial quality control during the 1920s and adapted to health sciences in the mid-1980s [29, 30] to manage units in a health system charged with delivering services, referred to in LQAS as supervision areas (SAs). In each SA, a random sample ('n') of individuals is assessed. A decision rule or cut-off value, designated as 'd', is selected to optimize identification of low performance SAs for a specified indicator. The 'd' depends on the sample size, thresholds for classifying high and low performance and the choice of two misclassification errors: the risk of overestimation when low coverage exists (β error) and the risk of underestimation when high coverage exists (α error). The upper threshold, 'p-Upper or pU', for identifying acceptably performing SAs is often suggested by

program stakeholders, and the low threshold, 'p-Lower or pL', is usually 30% lower.

are selected to ensure α-errors of ≤10%, and β-errors ≤ 10%—or more formally:

In the surveys used for this chapter, 19 locations were selected in each Network SA. The SA sample size, *n* = 19, guarantees that both misclassification errors do not exceed 0.10, as all corresponding values of 'pL' are 30% below 'pU'. SA having pL< p <Pu are classified according to their proximity to either pL or pU. SA with an intermediate performance has an equal chance of being classified as either high or low. The strength of LQAS is that it is very effective for identifying areas that are among the *worst of the worst* in terms of the indicator of interest. As an example, let us assume that the target for an indicator (such as exclusive breastfeeding) is set at 80%. The upper threshold, 'pU', is 80% for identifying high (or acceptably) performing SAs, while the lower threshold, 'pL', is 50%. The sample size 'n=19' and decision rule 'd=13'

The decision rules and error terms associated with varied sample sizes are found in [23, 24]. In Armenia, our intention was to apply LQAS for recurrent community monitoring by each of the three Network's members and the Network as a whole. The program area in each of the three sites was divided into administratively meaningful units or supervision areas. In each case, the Network member SAs were city areas, towns and surrounding villages. A supervisor —such as a doctor, or a nurse, or a midwife—managed each SA. With LQAS, data collectors

network in the capital under the leadership of ADRA.

88 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**2. Study methodology and results**

**•** P (X<d | n, pU ≥ 80%) ≤ α ≤ 0.10 **•** P (X≥d | n, pL ≤ 50%) ≤ β ≤ 0.10

**•** α + β< 0.20

**2.1. Sampling method**

A multi-stage sampling procedure was used to select the respondents. The first stage was a probability proportional to size (PPS) sampling method, meaning the larger the relative size of the community, the higher its chance of being selected. To carry out a PPS, a sampling frame was constructed, consisting of a list of all the communities in each SA with their population size. Each community population size is added to the next one to create a series of cumulative population sizes. So, if the first community has 453 people and the second one has 500, the first cumulative population size is 953 (or 453+500). Once all community population sizes have been summed, the sampling interval is calculated by dividing the total by the sample size (*n*=19 in this case). Assuming that the total population size is 43,518; when divided by 19 the result is 2290.42, which is the sampling interval. A random number between 1 and 2290 is selected using a random number table to identify the first of 19 random interview locations; for argument's sake, let us say this number is 1406. Subsequently, the sampling interval is added to the 1406 to identify the second interview location, which results in 1406 + 2290.42 = 3696.42.

The third location is 3696.42 + 2290.42 = 5986.84. This process is repeated until all 19 interview locations are identified. One then looks at the cumulative population size in the sampling frame and identifies the community in which the 1406th person resides and then where the 3696th person resides and so forth. This is done for each of the 19 numbers. An Excel spread sheet is often used to automate selection of the communities using PPS.

The second stage is a random selection of the starting household using segmentation sampling [31]. This was done either by using a government map of a community or a city area or by drawing the map of a community while in the field at the time of interviewing. The idea was to avoid having to draw a map of the entire community showing the location of each household. Instead, the community was divided into segments of approximately equal sizes using the landmarks.

The essential point is that each segment should have approximately the same number of households. All segments were then numbered, and one of them was randomly selected. If the selected segment was of a manageable size having 30 or fewer households, then the interviewers drew a detailed map of this segment depicting all households. If the selected segment was still large, it was further divided into sub-segments to reach a manageable population size. In the next step, the households were enumerated and one was randomly selected.

In the third stage, a respondent is selected in the household. The respondents in this study, which are referred to herein as our client group, belonged to populations associated with the content of the three questionnaires and linked with the main health services under assessment. Each client group was sampled independently in each SA and had *n*=19 interviews:

	- **◦** Antenatal care and safe delivery
	- **◦** Breastfeeding and child nutrition
	- **◦** Post-delivery family planning
	- **◦** Use of modern contraceptives
	- **◦** Abortion-seeking behaviour
	- **◦** Post-abortion information provision
	- **◦** Knowledge of danger signs during pregnancy, delivery, after delivery
	- **◦** Knowledge of danger signs in a newborn
	- **◦** Knowledge of HIV and preventative behaviour
	- **◦** Knowledge of danger signs during pregnancy, delivery, after delivery
	- **◦** Knowledge of danger signs in a newborn
	- **◦** Knowledge of HIV and preventive behaviour
	- **◦** Use of modern contraceptives

is 2290.42, which is the sampling interval. A random number between 1 and 2290 is selected using a random number table to identify the first of 19 random interview locations; for argument's sake, let us say this number is 1406. Subsequently, the sampling interval is added to the 1406 to identify the second interview location, which results in 1406 + 2290.42 = 3696.42.

The third location is 3696.42 + 2290.42 = 5986.84. This process is repeated until all 19 interview locations are identified. One then looks at the cumulative population size in the sampling frame and identifies the community in which the 1406th person resides and then where the 3696th person resides and so forth. This is done for each of the 19 numbers. An Excel spread sheet is

The second stage is a random selection of the starting household using segmentation sampling [31]. This was done either by using a government map of a community or a city area or by drawing the map of a community while in the field at the time of interviewing. The idea was to avoid having to draw a map of the entire community showing the location of each household. Instead, the community was divided into segments of approximately equal sizes

The essential point is that each segment should have approximately the same number of households. All segments were then numbered, and one of them was randomly selected. If the selected segment was of a manageable size having 30 or fewer households, then the interviewers drew a detailed map of this segment depicting all households. If the selected segment was still large, it was further divided into sub-segments to reach a manageable population size.

In the third stage, a respondent is selected in the household. The respondents in this study, which are referred to herein as our client group, belonged to populations associated with the content of the three questionnaires and linked with the main health services under assessment.

In the next step, the households were enumerated and one was randomly selected.

Each client group was sampled independently in each SA and had *n*=19 interviews:

**•** Women of reproductive age 15–49 years and not pregnant, interviewed to assess:

**◦** Knowledge of danger signs during pregnancy, delivery, after delivery

**•** Mothers with children 0–11 months, interviewed to assess:

**◦** Antenatal care and safe delivery

**◦** Breastfeeding and child nutrition

**◦** Post-delivery family planning

**◦** Use of modern contraceptives

**◦** Abortion-seeking behaviour

**◦** Post-abortion information provision

**◦** Knowledge of danger signs in a newborn

**◦** Knowledge of HIV and preventative behaviour

often used to automate selection of the communities using PPS.

90 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

using the landmarks.

For the purposes of this survey, a 'household' was defined as a group of persons who shared the same kitchen or hearth. A mother to be considered a member of the household should have lived in the household for at least 6 months. Any household could have representatives from more than one client group. In order to reduce inter-question correlations by sampling more than one sampling universe in the same household, the inclusion criterion stipulated that only one client group could be sampled in a given household. Therefore, interviewers were trained to delineate the composition of the household, and in case of more than one eligible person residing in the household, to select one for the interview using simple random sampling.

#### **2.3. Questionnaire development and interviewer training**

Each question in the questionnaire was associated with an indicator in the Network's M&E plan. Questions and option responses were taken from a field-tested Knowledge Practice and Coverage-2000 Instrument [32], which was adapted to local Network conditions. The instrument was translated into Armenian, back translated to English and then pre-tested in a suburb of Yerevan.

Training activities included a two-day training-of-trainers workshop in English, which took place in Yerevan, followed by three simultaneous five-day LQAS pre-survey training workshops in Armenian, which took place in Goris, Gavar and Gyumri. The workshops were supervised by a master trainer who was accompanied by a simultaneous translator from the Network and by members of the Network who were trained as LQAS trainers.

The data collection at each site began immediately following the five-day training and was supervised by a master trainer and trained members of the Network. Data collectors anticipated difficulty in finding eligible respondents due to two major concerns: first, many homes had been abandoned because of emigration; second, the fertility rate in Armenia was low, thereby limiting the number of households having women with babies less than one year old. Regardless of these concerns, the teams finished the data collection on schedule without major problems. Data collection in each SA took approximately 5 days, with two data collectors assigned to each SA.

#### **2.4. Data processing and analysis**

With LQAS, data can be analysed in two ways: through data hand tabulation and computer analysis. With hand tabulation, the master trainers lead the data collectors in a process of organizing the raw data taken directly from the questionnaires into tables to ascertain for each SA the number of individuals having the trait of interest. They use the decision rule to identify indicators, which have reached or not reached the stipulated coverage target. Computer statistical packages can also be used to achieve the same goal after data has been entered into prepared data screens.

While the choice of whether to tabulate data manually or to conduct computer analysis depends on the purpose of the survey, availability of resources, time constraints and the level of details required in the analysis, the two methods complement each other and can be used simultaneously. Each method has its advantages; the main advantage of hand tabulation is that it makes strategic information rapidly available to make urgent programmatic decisions within a couple of days after the survey. If the purpose of the survey is to quickly identify priorities among SAs and to immediately start addressing them, then hand tabulation is an appropriate choice. Computer data entry and analysis take longer to complete, as much as several weeks in many cases. But if the purpose of the survey is to get detailed results, including sociodemographic data, and to conduct complex analyses—for example, a test for association, or even to calculate 95% confidence intervals for each indicator and to weigh the data for differences in SA population size or socio-demographic groups—then a computer database is necessary.

For this survey, both hand tabulations and computer analyses were used. Data hand tabulation was facilitated by the LQAS trainer in each site and conducted by supervisors and data collectors during post-survey workshops. At the end of the workshops, the tables with tabulation results were produced and priority supervision areas for each indicator were identified in each catchment area. Later, data cleaning, computer data entry and analysis were conducted, producing 95% confidence intervals for each indicator and weighting the results for SA population size. Comparison of weighted and unweighted results revealed that the difference between them is small and, therefore, not essential for program planning.

To establish the reliability of the hand-tabulated data, we compared the hand-tabulated results with unweighted computer data. The total error for hand-tabulated data was 3% on average, which is acceptable.

#### **2.5. Identifying program priorities**

In the following sections, prevalence measures for key indicators will be presented that measure coverage of the population with health services in the three Network sites.

Network reproductive health program managers used these measures to identify *priority SAs* in their catchment areas, meaning the SAs that fell below average. In the case of ongoing annual monitoring of coverage, a target is used to assess an SA's performance, but as this is the first application of LQAS in Armenia, the average is used as p-Upper so as to establish program targets. The average is also applied to identify SAs that are outliers whose performance is particularly low and below average. When LQAS is done annually, it is the authors' view that using both the average coverage and the coverage target provides more complete information, an approach consistent with identifying SAs that are among the worst of the worst. Once the average coverage is calculated, a data collector uses the Composite LQAS Table (**Appendix 1**) to locate the column header corresponding to the average coverage. If the average is not divisible by 5, then it is rounded-up to the next highest value, which is divisible by 5 in order to use the LQAS table. Therefore, if, for example, the average coverage was 66%, it would round up to 70%. In the next step, the data collector locates the row for a sample of 19 (or the appropriate sample size if different from 19). At the intersection of this column and row, one finds the decision rule. All decision rules were determined using cumulative probabilities of the binomial model as explained in Section 1. If the total number of correct responses in an SA is less than the decision rule, then the SA is below average or did not reach the target and is in need of special attention. For example, if average coverage for an indicator was found to be 70%, then the decision rule would be 11. Any SA having less than 11 correct responses for that indicator would be judged to be below average.

**Appendix 2** demonstrates how data collectors used their data to make judgements about the SAs. It displays summary results for one of the program catchment area, Gyumri. The first indicator in this example is, 'Percentage of mothers attending an antenatal visit by a clinically trained provider.' In this catchment area, there were four SAs. The first series of columns shows the number of correct responses in each of the 4 SAs (13, 10, 13, 16), making the total correct 52. The next series of columns shows that the sample size in each SA was 19, for a total sample size of 76. The average coverage (68%) is calculated and recorded in the far right cell. The Composite LQAS Table was then used by data collectors to determine the decision rule using a rounded-up value of 70, resulting in a decision rule of 11. The highlighted cells indicate those SAs that were found to be below average. Two other indicators are included in this example to demonstrate how these LQAS data were used to identify priorities for local decision-making. The remainder of this chapter uses the aggregate measures only.

## **3. Results and discussion**

This section presents the results of the computer analysis of data from three types of respondents: non-pregnant women aged 15–49 years, men aged 15–54 years, and mothers of children aged 0–11 months. For the sake of simplicity, we refer to these three groups as women, men, and mothers.

The findings are shown for each project site as well as for the Armenia Network for Health as a whole, aggregating the results from the three pilot sites. The computer results are weighted by the population size of each supervision area. Qualitative data presented along with LQAS results complement the findings.

### **3.1. Family planning**

#### *3.1.1. Abortions*

indicators, which have reached or not reached the stipulated coverage target. Computer statistical packages can also be used to achieve the same goal after data has been entered into

While the choice of whether to tabulate data manually or to conduct computer analysis depends on the purpose of the survey, availability of resources, time constraints and the level of details required in the analysis, the two methods complement each other and can be used simultaneously. Each method has its advantages; the main advantage of hand tabulation is that it makes strategic information rapidly available to make urgent programmatic decisions within a couple of days after the survey. If the purpose of the survey is to quickly identify priorities among SAs and to immediately start addressing them, then hand tabulation is an appropriate choice. Computer data entry and analysis take longer to complete, as much as several weeks in many cases. But if the purpose of the survey is to get detailed results, including sociodemographic data, and to conduct complex analyses—for example, a test for association, or even to calculate 95% confidence intervals for each indicator and to weigh the data for differences in SA population size or socio-demographic groups—then a computer database is

For this survey, both hand tabulations and computer analyses were used. Data hand tabulation was facilitated by the LQAS trainer in each site and conducted by supervisors and data collectors during post-survey workshops. At the end of the workshops, the tables with tabulation results were produced and priority supervision areas for each indicator were identified in each catchment area. Later, data cleaning, computer data entry and analysis were conducted, producing 95% confidence intervals for each indicator and weighting the results for SA population size. Comparison of weighted and unweighted results revealed that the

To establish the reliability of the hand-tabulated data, we compared the hand-tabulated results with unweighted computer data. The total error for hand-tabulated data was 3% on average,

In the following sections, prevalence measures for key indicators will be presented that

Network reproductive health program managers used these measures to identify *priority SAs* in their catchment areas, meaning the SAs that fell below average. In the case of ongoing annual monitoring of coverage, a target is used to assess an SA's performance, but as this is the first application of LQAS in Armenia, the average is used as p-Upper so as to establish program targets. The average is also applied to identify SAs that are outliers whose performance is particularly low and below average. When LQAS is done annually, it is the authors' view that using both the average coverage and the coverage target provides more complete information, an approach consistent with identifying SAs that are among the worst of the worst. Once the average coverage is calculated, a data collector uses the Composite LQAS Table (**Appendix 1**) to locate the column header corresponding to the average coverage. If the average is not

measure coverage of the population with health services in the three Network sites.

difference between them is small and, therefore, not essential for program planning.

prepared data screens.

92 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

necessary.

which is acceptable.

**2.5. Identifying program priorities**

As **Table 1** shows, 44% of women reported a history of at least one induced abortion, with a TAR being 2.0 abortions per woman. A higher TAR of 2.6 in ADHS 2000 [33] may be attributed to the fact that our project sites included one small city (Gyumri) and two towns (Goris and Gavar) and did not include the national capital as well as other towns. The abortion rate varied across the three sites; Goris had a TAR of 1.4 abortions per woman, Gavar's TAR was 2.5, and Gyumri's was about 1.9. The ADHS 2000 results showed that on average an Armenian woman had 50% more abortions than births (2.6 TAR for 1.7 TFR). Further, comparison of the trends in TAR and TFR showed that although the TAR declined over the last 10 years (2.6 in 2000, 1.8 in 2005 and 0.8 in 2010), the total fertility rate remained stable over the same period (1.7), which is below replacement levels. The progressively fewer young woman becoming pregnant and giving birth, together with the overall reduced pregnancy rate [16] over time, may explain, at least in part, the decreasing TAR. Others suggest that with the introduction of medical abortions women practice self-administered abortions and do not necessarily seek medical assistance; as a result the total number of induced abortions is underreported [34]. At this point in its history, Armenia's health strategy needs to encourage and support families to wish to have more children, while also ensuring access to contraceptive methods and information, which will allow other couples to make informed choices and plan their family without an invasive procedure like abortion.


a Contraceptive prevalence rate (CPR) is the percentage of women who are currently using, or whose sexual partner is currently using, at least one method of contraception, regardless of the method used (WHO, http://www.who.int/ reproductivehealth/topics/family\_planning/contraceptive\_prevalence/en/). b CIs for separate sites are not indicated because of small sample sizes.

**Table 1.** Prevalence and 95%-confidence intervals of induced abortions, total abortion rate, family planning and postabortion information provision in the three sites of Armenia.

#### *3.1.2. Contraceptive method information and use*

across the three sites; Goris had a TAR of 1.4 abortions per woman, Gavar's TAR was 2.5, and Gyumri's was about 1.9. The ADHS 2000 results showed that on average an Armenian woman had 50% more abortions than births (2.6 TAR for 1.7 TFR). Further, comparison of the trends in TAR and TFR showed that although the TAR declined over the last 10 years (2.6 in 2000, 1.8 in 2005 and 0.8 in 2010), the total fertility rate remained stable over the same period (1.7), which is below replacement levels. The progressively fewer young woman becoming pregnant and giving birth, together with the overall reduced pregnancy rate [16] over time, may explain, at least in part, the decreasing TAR. Others suggest that with the introduction of medical abortions women practice self-administered abortions and do not necessarily seek medical assistance; as a result the total number of induced abortions is underreported [34]. At this point in its history, Armenia's health strategy needs to encourage and support families to wish to have more children, while also ensuring access to contraceptive methods and information, which will allow other couples to make informed choices and plan their family without an

**Pilot areas**

48% (±11%) 40% (±13%) 43% (±10%) 44% (±6%)

12% (±7%) 25% (±10%) 24% (±9%) 21% (±4%)

61% (±10%) 43% (±11%) 29% (±9%) 42% (±6%)

16% (±7%) 43% (±11%) 24% (±8%) 26% (±5%)

29% 45% 39% 37% (±9%)b

**Indicators Goris Gyumri Gavar Aggregate coverage**

 Contraceptive prevalence rate (CPR) is the percentage of women who are currently using, or whose sexual partner is currently using, at least one method of contraception, regardless of the method used (WHO, http://www.who.int/

**Table 1.** Prevalence and 95%-confidence intervals of induced abortions, total abortion rate, family planning and post-

Total abortion rate 1.40 1.85 2.52 2.03

)

reproductivehealth/topics/family\_planning/contraceptive\_prevalence/en/).

CIs for separate sites are not indicated because of small sample sizes.

abortion information provision in the three sites of Armenia.

invasive procedure like abortion.

94 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Percentage of women reporting an induced

Percentage of mothers counselled about the family planning after delivery

Percentage of women who report currently using a family planning method (CPRa

Percentage of women who report currently using a modern family planning method

**Post-abortion family planning**

a

b

Percentage of women counselled about family planning methods after abortion

**Abortions**

abortion in lifetime

**Contraceptive method information and use**

The contraceptive prevalence rate (CPRe3 ) was 42%, which is similar to the ADHS 2000 national estimate (39%). However, we detected substantial variation within the Network. Goris had the highest CPR (61%), and Gavar the lowest (29%), while the CPR in Gyumri approximated the Network average at 43% (**Figure 1**). About a quarter of all women used a modern method of contraception; however, there was significant variation among the sites. For example, Goris, the site with the highest CPR, had the smallest proportion of women using modern contraceptive methods (16%), indicating that most women preferred natural methods (e.g., withdrawal, abstinence, rhythm). In contrast, in Gavar, where the CPR was the lowest (29%), 75% of women using any family planning method used a modern method. Gyumri, as the second largest city in Armenia, had the highest rate of modern contraceptive use (43%), while having the second highest CPR among the three sites.

**Figure 1.** Contraceptive Prevalence Rate and Prevalence of Modern Family Planning Methods in the Three Sites of Armenia.

ADHS results show a slight decrease in the CPR (39% in 2000, 33% in 2005 and 34% in 2010) and variable use of modern contraceptives over a 10-year period (14% in 2000, 12% in 2005 and 17% in 2010) [35].

#### *3.1.3. Post-abortion family planning*

Further analysis of the survey results revealed that that only 37% of women who had had at least one induced abortion in their lifetime had been counselled regarding use of contraceptives after their abortion. A similar percentage (32%) was advised to use modern methods after the

<sup>3</sup> Contraceptive prevalence rate is the proportion of women of reproductive age who are using (or whose partner is using) a contraceptive method at a given point in time (WHO, 2006 http://www.who.int/whosis/whostat2006ContraceptivePrevalenceRate.pdf).

abortion, suggesting that those women who received any post-abortion counselling were likely to be informed about modern methods of family planning. One-third of women who had been exposed to post-abortion counselling reported using a modern method of contraception during the survey.

Postpartum experiences were also examined to determine whether mothers received family planning information. Only 21% of mothers reported being counselled about methods of contraception. While service providers play an influential role in the decision to use modern family planning methods in Armenia, some qualitative studies suggest that provider and client satisfaction with family planning counselling are critical factors [36, 37]. It remains unclear if service providers offer sufficient counsel to allay women's fears regarding modern contraceptives and to ensure that they feel empowered to make informed choices to use modern contraceptives.

According to these studies, the primary reasons for abortions were difficult socioeconomic conditions and a desire to postpone or stop childbearing. Among other factors affecting their decisions to abort, women cited insufficient effectiveness of contraceptives, lack of complete information about modern family planning methods, the side effects of the methods, difficulty of access, such as when women had to travel to the district centre to get contraceptives, and a lack of male motivation to use condoms. The views of service providers were also explored. The majority stressed the importance of providing adequate and comprehensive information and the need to discuss with women the variety of methods available. Some health care providers also emphasized the difference in health needs, level of education, preferences of women and respective importance of individual approach.

#### **3.2. Safe motherhood**

This section presents findings related to perinatal and newborn care. The respondents were women, men, or mothers, depending on the question. The questionnaire for mothers included both knowledge and behavioural questions, while the ones for women and men were knowledge questions. The outcomes thereof are summarized in **Tables 2** and **3**.

#### *3.2.1. Antenatal care*

Mothers with children 0–11 months were questioned about their pregnancy for their infant. Eighty percent of all mothers in the pilot sites said they had visited a clinically trained provider (gynaecologist, doctor, nurse, or midwife) at least once for antenatal care during that pregnancy. The ADHS 2000 found a slightly higher percentage (92%) for this indicator [38]. This variation is likely due to the Network study being confined to three locations, whereas the ADHS is nationally representative.

There was a variation among the sites. Gyumri had a lower percentage (67%) than Gavar (77%) and much lower than Goris (95%). Only 34% of all mothers interviewed had their first antenatal visit during the first trimester. The variation across the three sites for this indicator was similarly substantial (Goris: 45%; Gyumri: 26%; Gavar: 32%).

abortion, suggesting that those women who received any post-abortion counselling were likely to be informed about modern methods of family planning. One-third of women who had been exposed to post-abortion counselling reported using a modern method of contraception during

Postpartum experiences were also examined to determine whether mothers received family planning information. Only 21% of mothers reported being counselled about methods of contraception. While service providers play an influential role in the decision to use modern family planning methods in Armenia, some qualitative studies suggest that provider and client satisfaction with family planning counselling are critical factors [36, 37]. It remains unclear if service providers offer sufficient counsel to allay women's fears regarding modern contraceptives and to ensure that they feel empowered to make informed choices to use modern

According to these studies, the primary reasons for abortions were difficult socioeconomic conditions and a desire to postpone or stop childbearing. Among other factors affecting their decisions to abort, women cited insufficient effectiveness of contraceptives, lack of complete information about modern family planning methods, the side effects of the methods, difficulty of access, such as when women had to travel to the district centre to get contraceptives, and a lack of male motivation to use condoms. The views of service providers were also explored. The majority stressed the importance of providing adequate and comprehensive information and the need to discuss with women the variety of methods available. Some health care providers also emphasized the difference in health needs, level of education, preferences of

This section presents findings related to perinatal and newborn care. The respondents were women, men, or mothers, depending on the question. The questionnaire for mothers included both knowledge and behavioural questions, while the ones for women and men were knowl-

Mothers with children 0–11 months were questioned about their pregnancy for their infant. Eighty percent of all mothers in the pilot sites said they had visited a clinically trained provider (gynaecologist, doctor, nurse, or midwife) at least once for antenatal care during that pregnancy. The ADHS 2000 found a slightly higher percentage (92%) for this indicator [38]. This variation is likely due to the Network study being confined to three locations, whereas the

There was a variation among the sites. Gyumri had a lower percentage (67%) than Gavar (77%) and much lower than Goris (95%). Only 34% of all mothers interviewed had their first antenatal visit during the first trimester. The variation across the three sites for this indicator was

women and respective importance of individual approach.

96 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

similarly substantial (Goris: 45%; Gyumri: 26%; Gavar: 32%).

edge questions. The outcomes thereof are summarized in **Tables 2** and **3**.

the survey.

contraceptives.

**3.2. Safe motherhood**

*3.2.1. Antenatal care*

ADHS is nationally representative.


**Table 2.** Safe Motherhood: maternal care indicators and knowledge of maternal complications with 95%-confidence intervals in the three sites of Armenia.

Overall, a very low proportion of mothers (12%) reported receiving iron supplementation during their recent pregnancy, with Goris (6%) being significantly behind Gyumri and Gavar (17 and 12%, respectively). These results may cause concerns about maternal nutritional status. However, the ADHS 2000 found that only 12% of women in Armenia actually suffer from mild, moderate, or severe anemia [38].


**Table 3.** Indicators of newborn danger signs awareness with 95%-confidence intervals in the three sites of Armenia.

#### *3.2.2. Knowledge of danger signs*

Questions about danger signs during pregnancy, labour/delivery and postpartum were asked of both women and men. Men appeared to be substantially less knowledgeable than women about danger signs at any perinatal stage. Only 63% of women and 42% of men knew two or more danger signs during pregnancy. Gyumri and Goris reported the least knowledge for both men and women (women: 44 and 64%, respectively; men: 30 and 33%, respectively). Fifty-eight percent of women knew at least two danger signs during delivery. Again, Gyumri (43%) and Goris (47%) had lower levels of knowledge than Gavar (74%). Only 37% of men knew at least two danger signs during delivery, with the variation among pilot sites quite similar to the variation among women (**Table 2**).

Knowledge of two or more postpartum danger signs was demonstrated by 68% of women; consistent with other knowledge indicators, Gavar had the highest percentage of knowledgeable women (79%) followed by Goris (69%) and Gyumri (46%). Only 44% of men exhibited knowledge of postpartum danger signs, with similar results across the sites.

Overall knowledge of perinatal complications (knowing two or more danger signs during each of three stages: pregnancy, delivery and postpartum period) was low among both women (45%) and men (23%). Gavar exhibited relatively higher level of knowledge than the other two sites: 58% of the women knew two more danger signs during each of the three stages. Nevertheless, the WHO data show a slow but stable decrease in the maternal mortality ratio in Armenia, from 40 in 2000 and 2005 to 33 in 2010 and 25 in 2015 [39]. This may be due to emergency care in Armenia being accessible or to the increasing awareness of men and women to perinatal danger signs. However, the ADHS 2010 did not include this information and we cannot substantiate our view regarding the declining MMR.

#### *3.2.3. Newborn care*

Women and men were asked about newborn danger signs within the first 7 days of birth. Seventy percent of women knew two or more danger signs of a newborn. As with maternal care, the level of knowledge in Gyumri was lower (52%), than in Goris (72%) and Gavar (78%) (**Table 3**).

Among men, 50% knew two or more danger signs in a newborn in the first 7 days of birth. The pattern of variation is similar to that for women, with Gyumri having a lower level of knowledge (42%) than Gavar and Goris (52 and 55%, respectively).

#### **3.3. Breastfeeding and complementary feeding**

**Indicator Pilot areas Aggregate coverage**

**Table 3.** Indicators of newborn danger signs awareness with 95%-confidence intervals in the three sites of Armenia.

Questions about danger signs during pregnancy, labour/delivery and postpartum were asked of both women and men. Men appeared to be substantially less knowledgeable than women about danger signs at any perinatal stage. Only 63% of women and 42% of men knew two or more danger signs during pregnancy. Gyumri and Goris reported the least knowledge for both men and women (women: 44 and 64%, respectively; men: 30 and 33%, respectively). Fifty-eight percent of women knew at least two danger signs during delivery. Again, Gyumri (43%) and Goris (47%) had lower levels of knowledge than Gavar (74%). Only 37% of men knew at least two danger signs during delivery, with the variation among pilot sites quite similar to the

Knowledge of two or more postpartum danger signs was demonstrated by 68% of women; consistent with other knowledge indicators, Gavar had the highest percentage of knowledgeable women (79%) followed by Goris (69%) and Gyumri (46%). Only 44% of men exhibited

Overall knowledge of perinatal complications (knowing two or more danger signs during each of three stages: pregnancy, delivery and postpartum period) was low among both women (45%) and men (23%). Gavar exhibited relatively higher level of knowledge than the other two sites: 58% of the women knew two more danger signs during each of the three stages. Nevertheless, the WHO data show a slow but stable decrease in the maternal mortality ratio in Armenia, from 40 in 2000 and 2005 to 33 in 2010 and 25 in 2015 [39]. This may be due to emergency care in Armenia being accessible or to the increasing awareness of men and women to perinatal danger signs. However, the ADHS 2010 did not include this information and we

Women and men were asked about newborn danger signs within the first 7 days of birth. Seventy percent of women knew two or more danger signs of a newborn. As with maternal care, the level of knowledge in Gyumri was lower (52%), than in Goris (72%) and Gavar (78%)

knowledge of postpartum danger signs, with similar results across the sites.

cannot substantiate our view regarding the declining MMR.

Percentage of women knowing 2 or more danger signs in newborns within first 7 days of birth

98 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Percentage of men knowing two or more danger signs in newborn with first 7 days of birth

*3.2.2. Knowledge of danger signs*

variation among women (**Table 2**).

*3.2.3. Newborn care*

(**Table 3**).

**Goris Gyumri Gavar**

72% (±9%) 52% (±11%) 78% (±8%) 70% (±6%)

55% (±10%) 42% (±11%) 52% (±10) 50% (±6)

The questionnaire for mothers of children 0–11 months included questions about initiating breastfeeding, exclusive breastfeeding and introduction of complementary foods (**Table 4**). For some indicators, the data were analysed for subgroups rather than the whole sample. For example, the assessment of *exclusive breastfeeding* included mothers of children ages 0–5 months, while the assessment of *complementary feeding* practices included mothers of children ages 6–9 months. Other indicators included the full sample of women with children 0–11 months.


**Table 4.** Breastfeeding indicators with 95%-confidence intervals among mothers of children 0-11 months in the three sites of Armenia.

Twenty-eight percent of mothers began breastfeeding newborns within 1 h of the birth; values ranged from 3% in Goris to 41% in Gavar. Gyumri, at 31%, was about average. The low percentage of newborns breastfeeding within the first hour of life could be attributed to the practice of separating newborns from their mothers immediately after delivery. Less than onethird of mothers (30%) reported that their babies were given to them right after the cutting of the umbilical cord. Overall, less than a quarter of mothers with children aged 0–5 months (23%) reported exclusive breastfeeding at the time of the survey. In a smaller sub-sample of mothers with children aged 0–3 months, the proportion of mothers who exclusively breastfed was 34%. This result indicated that mothers commenced giving liquids or complementary feeding earlier than recommended.

A trend analysis of exclusive breastfeeding indicates a rapid decline in the practice by the second month of life, and it continues to decline steeply until 4 months of age (**Figure 2**). While two-thirds of mothers (66%) reported they were exclusively breastfeeding their 0–1-month-old babies, the prevalence of exclusive breastfeeding decreased with a child's age, with only 43, 22 and 8% of mothers breastfeeding their babies exclusively in the second, third and fourth months, respectively. By the fifth month, no mothers reported they were exclusively breastfeeding their babies.

**Figure 2.** Infant feeding in a cohort of infants aged 0–11 months.

The majority (85%) of mothers of children ages 6–9 months reported that they were giving complementary foods to their babies. More than 90% of mothers in Goris and Gavar fed their 6–9-month-old babies complementary foods, but in Gyumri, only 74% of mothers reported giving complementary foods. Consistently, a high proportion of mothers were giving complementary food to their 6–9-month-old children for each month of age. A trend analysis for complementary feeding showed that 74% of the mothers were practicing complementary feeding at 6 months and 84% by 9 months. When the 'food' category was disaggregated into liquids and solids and all mothers with children ages 0–11 months were included, the earliest premature introduction of solids occurred at 1 month and was 18% by the third month, while 34% of mothers began giving their infants liquids during the first month of life. In ADHS 2010, 23% of mothers of children 2–3 months gave their babies non-milk liquids in addition to breastfeeding, and 14% gave other milk in addition to breastfeeding; 35% of mothers gave complementary foods to their babies of 4–5 months. The tendency to introduce both liquids and solids at an early age, by as early as 1 month, and the consequent rapid decline in exclusive breastfeeding is an area for program action at the present time.

## **4. Current research priorities in areas of reproductive health in Armenia**

A trend analysis of exclusive breastfeeding indicates a rapid decline in the practice by the second month of life, and it continues to decline steeply until 4 months of age (**Figure 2**). While two-thirds of mothers (66%) reported they were exclusively breastfeeding their 0–1-month-old babies, the prevalence of exclusive breastfeeding decreased with a child's age, with only 43, 22 and 8% of mothers breastfeeding their babies exclusively in the second, third and fourth months, respectively. By the fifth month, no mothers reported they were exclusively breast-

The majority (85%) of mothers of children ages 6–9 months reported that they were giving complementary foods to their babies. More than 90% of mothers in Goris and Gavar fed their 6–9-month-old babies complementary foods, but in Gyumri, only 74% of mothers reported giving complementary foods. Consistently, a high proportion of mothers were giving complementary food to their 6–9-month-old children for each month of age. A trend analysis for complementary feeding showed that 74% of the mothers were practicing complementary feeding at 6 months and 84% by 9 months. When the 'food' category was disaggregated into liquids and solids and all mothers with children ages 0–11 months were included, the earliest premature introduction of solids occurred at 1 month and was 18% by the third month, while 34% of mothers began giving their infants liquids during the first month of life. In ADHS 2010, 23% of mothers of children 2–3 months gave their babies non-milk liquids in addition to breastfeeding, and 14% gave other milk in addition to breastfeeding; 35% of mothers gave complementary foods to their babies of 4–5 months. The tendency to introduce both liquids and solids at an early age, by as early as 1 month, and the consequent rapid decline in exclusive

feeding their babies.

**Figure 2.** Infant feeding in a cohort of infants aged 0–11 months.

100 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

breastfeeding is an area for program action at the present time.

In accordance with the statement of the International Conference on Population and Development Program of Action, "reproductive health… implies that people are able to have a responsible, satisfying and safe sex life and that they have the capability to reproduce and the freedom to decide if, when and how often to do so…The aim of interventions is to enhance reproductive health and promote reproductive rights rather than population policies and fertility control" [40].

The results of ADHS 2010 showed that 21% of married women of reproductive age had an unmet need for contraceptives, while the World Bank's data indicated a lower prevalence of 13.5% for the same year [41]. Another source reported this indicator as 19% [42]. In either case, this result indicates that access to modern birth spacing methods and complete information on contraceptives might be still an issue, although there are also other priorities for reproductive health research and policy in Armenia. Infertility and termination of pregnancies due to socio-economic difficulties is now evident and needs further investigation. Fifteen percent of women in ADHS 2010 reported termination of pregnancy due to socio-economic difficulties. Eight percent of women of reproductive age declared being infecund in 2010 as compared to four percent of women in 2005 [43].

Types and causes of infertility need investigation, and prevention and treatment programs should be designed, implemented and evaluated. A law to support families with more than one child should be introduced, its implementation should be tracked, and its effectiveness should be assessed. The low status of women and various socioeconomic barriers contribute to the high level of sex-selective abortions, indicating that programs to empower women and to support families wanting more children need to be implemented and monitored. In all community assessments, the LQAS methodology can be used. Using the modification of LQAS developed for large countries makes it possible to generate statistically rigorous results on both national and regional levels and at an even lower relative low cost as compared to conventional cluster sampling [44].

This observation is not intended to denigrate DHS and MICS, but we believe it is appropriate and preferable to use rapid and affordable methods for recurrent local-level monitoring that supports sound program management. LQAS can be applied not only to survey the community but also to assess health facilities (HFA). Using the LQAS for HFA allows detecting the gaps in quality of health care provision and identifying priorities for resource allocation and systematic monitoring [45–47].

During 2000 when this survey was undertaken, abortion was already an outmoded means of family planning. Today, 15 years later, this is even more true and yet the current TAR in Armenia signals the need for ongoing local level assessments of its prevalence and cause. The results of this study and the methodology used may support the development of sound health policies for Armenia's and other nations.

## **Appendices**


N/A: *not applicable*, meaning LQAS cannot be used in this assessment because the coverage is either too.

unshaded cells indicate where alpha or beta are < 10%

shaded cells indicate where *alpha* or *beta* errors are ≥ 10%.

hashed cells indicate where *alpha* or *beta* errors are > 15%.

**Appendix 1:** Composite LQAS Table.


**Appendix 2.** A sample of hand tabulated data—summary results for mothers with children 0–11 months—baseline survey.

## **Author details**

**Appendices**

**LQAS Table: Decision rules for sample sizes of 12–30 and coverage targets/average of 10–95% Sample size\* Average coverage (baselines)/annual coverage target (monitoring & evaluation)**

102 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%**

12 N/A N/A 1 1 2 2 3 4 5 5 6 7 7 8 8 9 10 11

13 N/A N/A 1 1 2 3 3 4 5 6 6 7 8 8 9 10 11 11

14 N/A N/A 1 1 2 3 4 4 5 6 7 8 8 9 10 11 11 12

15 N/A N/A 1 2 2 3 4 5 6 6 7 8 9 10 10 11 12 13

16 N/A N/A 1 2 2 3 4 5 6 7 8 9 9 10 11 12 13 14

17 N/A N/A 1 2 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 N/A N/A 1 2 2 3 5 6 7 8 9 10 11 11 12 13 14 16

19 N/A N/A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

20 N/A N/A 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17

21 N/A N/A 1 2 3 4 5 6 8 9 10 11 12 13 14 16 17 18

22 N/A N/A 1 2 3 4 5 7 8 9 10 12 13 14 15 16 18 19

23 N/A N/A 1 2 3 4 6 7 8 10 11 12 13 14 16 17 18 20

24 N/A N/A 1 2 3 4 6 7 9 10 11 13 14 15 16 18 19 21

25 N/A 1 2 2 4 5 6 8 9 10 12 13 14 16 17 18 20 21

26 N/A 1 2 3 4 5 6 8 9 11 12 14 15 16 18 19 21 22

27 N/A 1 2 3 4 5 7 8 10 11 13 14 15 17 18 20 21 23

28 N/A 1 2 3 4 5 7 8 10 12 13 15 16 18 19 21 22 24

29 N/A 1 2 3 4 5 7 9 10 12 13 15 17 18 20 21 23 25

30 N/A 1 2 3 4 5 7 9 11 12 14 16 17 19 20 22 24 26

N/A: *not applicable*, meaning LQAS cannot be used in this assessment because the coverage is either too.

 unshaded cells indicate where alpha or beta are < 10% shaded cells indicate where *alpha* or *beta* errors are ≥ 10%. hashed cells indicate where *alpha* or *beta* errors are > 15%.

**Appendix 1:** Composite LQAS Table.

Joseph J. Valadez\* and Lusine Mirzoyan

\*Address all correspondence to: joseph.valadez@lstmed.ac.uk

Liverpool School of Tropical Medicine, Liverpool, UK

## **References**

[1] Family planning/Contraception: Fact sheet N°351. World Health Organization, WHO, 2015. http://who.int/mediacentre/factsheets/fs351/en/


[16] Armenia Demographic and Health Survey. 2010. National Statistical Service Armenia, Ministry of Health Armenia, ICF International, USA. 2011.

[2] Facts on induced abortions worldwide: Fact sheet. Guttmacher Institute, USA. May, 2016. http://www.who.int/reproductivehealth/publications/unsafe\_abortion/abor-

[3] Say L, Chou D, Gemmill A, Tunçalp Ö, Moller A, Daniels J, Gülmezoglu M, Temmerman M, Alkema L.Global causes of maternal death: a WHO systematic analysis. The

[4] Black R, Laxminarayan R, Temmerman M, Walker N. Reproductive, Maternal, Newborn, and Child Health. Disease Control Priorities, Washington, DC, World Bank, 2016.

[5] Ethical issues in obstetrics and gynecology. FIGO Committee for the Study of Ethical

[6] Gissler M, Kauppila R, Meriläinen J, Toukomaa H, Hemminki E. Pregnancy-associated deaths in Finland 1987–1994: definition problems and benefits of record linkage. Acta

[7] Reardon D, Coleman P. Short and long term mortality rates associated with first pregnancy outcome: population register based study for Denmark 1980–2004. Medical

[8] Parazzini F, Vecchia C, Negri E, Checchetti G, Fedele I. Reproductive factors and the risk of invasive and intraepithelial cervical neoplasia. British Journal of Cancer.

[9] Remennick L. Induced abortion as cancer risk factor: a review of epidemiological evidence. Journal of Epidemiology and Community Health. 1990;44:259-264.

[10] Weiss N. Events of reproductive life and the incidence of epithelial ovarian cancer.

[11] Vecchia C, Negri E, Franceschi S, D'Avanzo B. Reproductive factors and the risk of hepatocellular carcinoma in women. International Journal of Cancer. 1992;52:351.

[12] Induced abortion operations and their early sequelae. Journal of the Royal College of

[13] Freedman M. Comparison of complication rates in first trimester abortions performed by physician assistants and physicians. American Journal of Public Health. 1986:76(5):

[14] OECD. Development in Eastern Europe and the South Caucasus: Armenia, Azerbaijan, Georgia, Republic of Moldova and Ukraine. OECD Publishing, 2011. http://dx.doi.org/

[15] Armenia Demographic and Health Survey. 2000. National Statistical Service Armenia,

Aspects of Human Reproduction and Women's Health; 2012.

Obsetricia et Gynecolgica Scandinavica. 1997;76:651-657.

American Journal of Epidemiology. 1983;117(2):128-139.

General Practitioners.1985;35(73):175-180.

Ministry of Health Armenia, ORC Macro, USA.

tion\_facts/en/

Lancet. 2014;2(6):e323–e333.

Third Edition (Volume 2): Chapter 3.

104 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Science Monitor. 2012;18(9):PH71-6.

1989;59:805-809.

550-554.

10.1787/9789264113039-en


[42] Alcema L, Kantorova V, Menozzi C, Biddlecom A. National, regional, and global rates and trends in contraceptive prevalence and unmet need for family planning between 1990 and 2015: a systematic and comprehensive analysis. The Lancet. 2013;381(9878): 1647. Available from: http://dx.doi.org/10.1016/S0140-6736(12)62204-1 [accessed 2016-09-03].

[29] Robertson SE, Anker M, Roisin AJ, Macklai N, Engstrom K. The lot quality technique: a global review of applications in the assessment of health services and diseases

[30] Reinke WA. Industrial Sampling Plans: Prospects for Public Health Applications. Occasional Paper. Baltimore: Institute for International Programs, The Johns Hopkins

[31] Turner AG, Magnani RJ, Shuaib M. A not quite as quick but much cleaner alternative to the Expanded Program on Immunization (EPI) Cluster Survey design. International

[32] CORE Group. Knowledge, Practice and Coverage Survey –2000+. Washington, DC, October, CORE Group, 2000. Available from: http://pdf.usaid.gov/pdf\_docs/

[33] Saribekyan K, Abrahamyan R, Balasanyan M, Hovhannisyan A. Maternal and child

[34] Jilozian A, Agadjanian V. Is induced abortion really declining in Armenia? Studies in

[35] Westoff C. A new approach to estimating abortion rates: DHS analytical studies 13.

[36] Thompson M, Harutyunyan T, Ghukasyan G: Feasibility Study: The Strategic Introduction of the Standard Days Method of Family Planning in Armenia: Formative Research Final Report. Yerevan, Armenia: American University of Armenia, The Center for Health Services Research and Georgetown University, Institute for Reproductive

[37] Salvador S, Danielian L. Report on Qualitative Research: JHU/PCS Project on Reproductive Health in Armenia. Yerevan, Armenia: American University in Armenia, The

[38] Reproductive, maternal and child health in Eastern Europe and Eurasia: a comparative report. CDC, ORC Macro DHS. Atlanta, USA. 2003; Chapter 12: p 156,157.

[39] Maternal Mortality in 1990–2015: Armenia. WHO, UNICEF, UNFPA, World Bank Group, and United Nations Population Division Maternal Mortality Estimation Inter-Agency Group. Geneva, World Health Organization: 2015. Available from: http://

[40] Guidelines on Reproductive Health. United Nations Population Information Network (POPIN). Available from: http://www.un.org/popin/unfpa/taskforce/guide/iatfreph.

[41] The World Bank Group. 2016. Available from: http://data.worldbank.org/indicator/

www.who.int/gho/maternal\_health/countries/arm.pdf [accessed 2016-08-15]

Center for Health Services Research and The Center for Policy Analysis; 1999.

health. National Statistical Service of the RA. 2000; Chapter 12: p 125.

Family Planning. 2016;47(2):163-78. doi:10.1111/j.1728-4465.2016.00053.x.

surveillance. World Health Statistical Quarterly. 1997;50:199-209.

University School of Hygiene and Public Health, 1988.

Journal of Epidemiology. 1996;25(1):198-203.

106 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Macro International, USA; 2008, p 8.

gdl.html [accessed 2016-07-05]

SP.UWT.TFRT?locations=AM [accessed 2016-09-03].

Pnack209.pdf.

Health; 2001.


**Quality Frameworks and Management**

#### **A Framework to Manage Quality of Enterprise Content Management Systems A Framework to Manage Quality of Enterprise Content Management Systems**

José González Enríquez, Francisco José Domínguez Mayo, Julián Alberto García García, María José Escalona Cuaresma and Manuel Mejías Risoto José González Enríquez, Francisco José Domínguez Mayo, Julián Alberto García García, María José Escalona Cuaresma and Manuel Mejías Risoto

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66199

#### **Abstract**

There is a wide range of enterprise content management (ECM) systems which supports, among other things, document management processes, records management and Web content management. However, each of these systems has many features and some of them can meet organizational needs depending on the scale, sector and workflow of the organization. In addition, it is very common that organizations are unaware of what ECM system best fits their needs, since each company has its particular scope and strategic objectives. This chapter is contextualized within the real project called THOT designed for the Andalusian Public Administration in Spain. The aim of this project is to study in detail ECM systems and propose an objective method to compare them for the specific scope and strategic objective of organizations. Quality evaluation frame‐ work (QuEF) has been adapted for this purpose.

**Keywords:** software engineering, standards, methodologies, enterprise content management, quality analysis and evaluation

## **1. Introduction**

Today's world economic situation is dominated by concepts such as globalization, which involves the relocation of companies, the constant search for lower costs to maximize profits

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

by continuous mergers and acquisitions, and a constant motivation for improving and optimizing all processes. Defining business models to manage organizations effectively is essential and nowadays, it has become a common practice followed by a large number of companies in all areas of business, especially within the information and communications technologies (ICT) context [1]. This can be extrapolated to the context of document processes, as these can help sales and product development organizations capitalize on new business opportunities [2]. For example, this implementation would permit a faster commercialization of new offerings, a flexible response to customers' needs, a quicker reply to changing market dynamics and business competitiveness improvement.

Every day, organizations have to handle a lot of different type of information such as general documents, audio‐visual files/resources, management reports, customer cards or invoices, among others. In addition, this information is not disconnected, but integrated within the organization's business process. The ever‐growing use of this kind of information within ICT organizations confirms this tendency for the years to come. There are two main alternatives to manage efficiently and effectively these organizations' information resources. The first one consists in specifying, designing and implementing an ad‐hoc system for this goal within organizations. This alternative is not usually feasible in most of them due to the high cost of maintenance, development and future updates.

The second option deals with using proprietary or open source enterprise content management (ECM) solutions. Performed literature review indicates that this option best fits companies' budgets.

A variety of ECM definitions can be found in the literature, according to the Association for Information and Image Management (AIIM) [3]. The ECM consists of strategies, methods and systems used to capture, manage, store, preserve and deliver content and documents related to organizational processes. ECM systems (ECMSs) and strategies facilitate an organization's unstructured information management. However, there currently is a wide range of document management solutions with different cost and functional scope (such as interoperability, security, usability, efficiency, customization among others). Therefore, to choose the one which best meet the organization's necessities is not a simple task. Smith and McKeen [4] define ECM solutions as an integrated approach to manage all organization's information including paper documents, data, reports, Web pages and digital assets. In addition, Tyrväinen et al. [5] provide a framework to stimulate and guide future research as well as point out research issues specific to the ECM field. The aforementioned authors argue that ECM offers an important and complex subfield of information systems. As regards the term ECM, these authors add that it has been widely adopted by software product vendors and practitioners in order to refer to technologies used to manage the content of assets like documents, Web sites, intranets and extranets in organizational or inter‐organizational contexts. Preliminary findings suggest that ECM has not managed to attract adequate attention.

Nordheim et al. [6] also examine the topic of ECM solutions, but under a strategic development and implementation process for a large oil company. The authors represent a case of a hybrid development approach to ECM that involves the lifecycle, teleological and dialectical engines of development. They state that this is opposite to the evolutionary development motor, which has prevailed in the hitherto reported content management research. Nordheim et al. also show a case study which complements process‐based research on enterprise system implementa‐ tions in general. In addition, they suggest that research and practice on large‐scale ECM implementations should acknowledge all the four motors of change.

by continuous mergers and acquisitions, and a constant motivation for improving and optimizing all processes. Defining business models to manage organizations effectively is essential and nowadays, it has become a common practice followed by a large number of companies in all areas of business, especially within the information and communications technologies (ICT) context [1]. This can be extrapolated to the context of document processes, as these can help sales and product development organizations capitalize on new business opportunities [2]. For example, this implementation would permit a faster commercialization of new offerings, a flexible response to customers' needs, a quicker reply to changing market

Every day, organizations have to handle a lot of different type of information such as general documents, audio‐visual files/resources, management reports, customer cards or invoices, among others. In addition, this information is not disconnected, but integrated within the organization's business process. The ever‐growing use of this kind of information within ICT organizations confirms this tendency for the years to come. There are two main alternatives to manage efficiently and effectively these organizations' information resources. The first one consists in specifying, designing and implementing an ad‐hoc system for this goal within organizations. This alternative is not usually feasible in most of them due to the high cost of

The second option deals with using proprietary or open source enterprise content management (ECM) solutions. Performed literature review indicates that this option best fits companies'

A variety of ECM definitions can be found in the literature, according to the Association for Information and Image Management (AIIM) [3]. The ECM consists of strategies, methods and systems used to capture, manage, store, preserve and deliver content and documents related to organizational processes. ECM systems (ECMSs) and strategies facilitate an organization's unstructured information management. However, there currently is a wide range of document management solutions with different cost and functional scope (such as interoperability, security, usability, efficiency, customization among others). Therefore, to choose the one which best meet the organization's necessities is not a simple task. Smith and McKeen [4] define ECM solutions as an integrated approach to manage all organization's information including paper documents, data, reports, Web pages and digital assets. In addition, Tyrväinen et al. [5] provide a framework to stimulate and guide future research as well as point out research issues specific to the ECM field. The aforementioned authors argue that ECM offers an important and complex subfield of information systems. As regards the term ECM, these authors add that it has been widely adopted by software product vendors and practitioners in order to refer to technologies used to manage the content of assets like documents, Web sites, intranets and extranets in organizational or inter‐organizational contexts. Preliminary findings suggest that

Nordheim et al. [6] also examine the topic of ECM solutions, but under a strategic development and implementation process for a large oil company. The authors represent a case of a hybrid development approach to ECM that involves the lifecycle, teleological and dialectical engines of development. They state that this is opposite to the evolutionary development motor, which

dynamics and business competitiveness improvement.

112 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

maintenance, development and future updates.

ECM has not managed to attract adequate attention.

budgets.

Furthermore, Munkvoldet et al. [7] explain that the concept of ECM represents integrated enterprise‐wide management of the lifecycles of all forms of recorded information content and their metadata, organized according to corporate taxonomies and supported by appropriate technological and administrative infrastructures. This publication is based on a case study about a Norwegian oil company (Statoil), where they identify a wide range of issues related to content, infrastructure and change management. As ECM perspective is concerned, Munkvoldet et al. argue that ECM perspective is found to integrate and extend the existing research areas of information resource and document management, as well as the repository model of knowledge management. Thus, they show how ECM deserves further attention beyond its current market hype, as a potential area of Information Systems (IS) research crossing several previously separate areas of information management from the enterprise viewpoint.

All ECM definitions show that there is a wide range of ECM descriptions and solutions in literature which support, among other things, document management processes, record management and Web content management. The problem is that each of these solutions has many features and some of them can meet organizational needs depending on the scale, sector and workflow of the organization. In addition, it is very common that organizations are unaware of what ECM system best fits their needs because each company has its particular scope and strategic objectives.

This chapter is contextualized within the THOT project, which is a real project designed for the Andalusian Public Administration in Spain. Document processes management is essential and critical in this context, since e‐Government is taking a key role in setting the strategic plans of the Public Administration. Therefore, the AOPJA (Public Agency of Contracting Services for Transport and Infrastructure Constructions) and the University of Seville are executing the THOT project as a result of these needs. THOT is an innovative project with high costs (621,250 Euros) focused on document management applied to service agreement records and transport infrastructure projects. Consequently, deciding on the most suitable ECM solution for the AOPJA context poses considerable responsibility. An ECM system that meets all requirements must be developed once the most suitable ECM solution is characterized and defined for the AOPJA context.

Therefore, THOT project aims to analyse in detail strategies and document management systems to investigate and define an innovative solution, so that records management can improve. Then, the objective is to study ECM systems in detail and propose an objective method to compare them for the specific scope and strategic objective of the AOPJA.

For this purpose, a quality evaluation framework (named QuEF) is proposed to analyse and evaluate ECM solutions. QuEF is a work‐in‐progress framework that has been used in a case of study (Public Administration of the Regional Government of Andalusia, Spain) in order to validate it. Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, allowing organizations to choose the best ECM system for their necessities and ensure the continuous quality improvement of these systems in the organization.

For these reasons, the objective of this chapter is twofold: to identify assessment criteria for ECM solutions, focused on the AOPJA context, by means of the proposed framework and to demonstrate the value QuEF offers to support technology acquisition.

As such, this work is structured as follows: Section 2 summarizes some of the most recent work related to ECM systems and reference standards on this topic.

Section 3 describes QuEF and its theoretical foundations and Section 4 presents the THOT project.

Then, Section 5 explains how the framework has been applied to the project and introduces each phase of the framework and its execution process along the project. Finally, Section 6 states learned lessons and on‐going work.

## **2. Related work**

There are different versions of ECM systems found in literature; Scott [8] evaluates the factors that lead to users choosing an ECM system. The results show the importance of cognitive involvement with technology highlighting the importance of including cognitive participation in building acceptance studies.

Alalwan and Weistroffer [9] performed a comprehensive literature review of ECM. A concep‐ tual framework of the areas of interest in relation to ECM as well as an agenda for future research on this topic were proposed.

Ninety‐one ECM publications were reviewed in this work. The authors concluded that the ECM systems involve interacting technical, social, organizational and business aspects. Also, the authors suggested that the current ECM literature could be grouped into three main research pylons:


Smith and McKeen include the strategies, tools, processes and skills an organization needs to manage its information assets on ECM solutions along its lifecycle. The authors explained that an effective ECM strategy should address each of the four lifecycles stages:


validate it. Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, allowing organizations to choose the best ECM system for their necessities and

For these reasons, the objective of this chapter is twofold: to identify assessment criteria for ECM solutions, focused on the AOPJA context, by means of the proposed framework and to

As such, this work is structured as follows: Section 2 summarizes some of the most recent work

Section 3 describes QuEF and its theoretical foundations and Section 4 presents the THOT

Then, Section 5 explains how the framework has been applied to the project and introduces

There are different versions of ECM systems found in literature; Scott [8] evaluates the factors that lead to users choosing an ECM system. The results show the importance of cognitive involvement with technology highlighting the importance of including cognitive participation

Alalwan and Weistroffer [9] performed a comprehensive literature review of ECM. A concep‐ tual framework of the areas of interest in relation to ECM as well as an agenda for future

Ninety‐one ECM publications were reviewed in this work. The authors concluded that the ECM systems involve interacting technical, social, organizational and business aspects. Also, the authors suggested that the current ECM literature could be grouped into three main

**•** The final one constitutes the strategic managerial aspect. An agenda for future based on the

Smith and McKeen include the strategies, tools, processes and skills an organization needs to manage its information assets on ECM solutions along its lifecycle. The authors explained that

**2.** Organize: indexing, classifying and linking content and databases together to provide

ensure the continuous quality improvement of these systems in the organization.

demonstrate the value QuEF offers to support technology acquisition.

each phase of the framework and its execution process along the project.

related to ECM systems and reference standards on this topic.

114 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Finally, Section 6 states learned lessons and on‐going work.

**•** The first one consists of the four ECM component dimensions.

review and the suggested conceptual framework is also suggested.

an effective ECM strategy should address each of the four lifecycles stages:

**1.** Capture: gathering all activities associated with collecting content.

access within and across business units and functions.

**•** The second one deals with the enterprise system lifecycle.

project.

**2. Related work**

research pylons:

in building acceptance studies.

research on this topic were proposed.

The authors pointed out that most ECM initiatives take a bottom‐up approach that focuses on delivering immediate benefits through projects such as intranet portals, information searching and Web content management, while the top‐down vision for ECM includes improved decision‐making, better utilization of information and collection of competitive intelligence. However, performed research indicated that managers also acknowledge the fact that greater value can be gained from taking a more strategic approach to ECM. The authors showed that those organizations that can effectively "hand‐shake" content stewardship practices with appropriate information behaviours, and values and information technology on a broader scale, can have a significant effect on their performance.

Grahlmann et al. [10] explain that ECM centers on managing all types of content used in organizations. The authors present an overview of previous research explaining that scientific literature on ECM is limited and no consensus on the definition of ECM is reached. Therefore, the literature review surfaces several ECM definitions that are merged herein into a more consistent and comprehensive definition of ECM. Indeed, the authors mentioned above provide the functional ECM framework (FEF) which is an overview of the potential function‐ alities of ECMSs. They applied FEF to three case studies to communicate on ECMSs, to familiarize oneself with and to direct future research. It may also form the basis for more formal reference architecture, and practitioners may use it as an assessment tool for comparing the functionalities provided by existing ECMSs.

Herbst et al. [11] showed that ECM is an important enabler of informatioey identified a set of critical success factors for ECM and develop a framework that helps organizations assess their readiness for ECM. In Herbst et al., this framework was developed following the data collected in workshops held between ECM project leaders and members of five companies. The authors argued that expert's opinions and experiences are combined with research results from the academic literature, and two illustrative cases showed how the framework has been put into practice.

In addition, Rickenberg et al. [12] explained that ECM can be considered an integrated approach to information management. They exposed that ECM research is still an emerging field of IS research, even though practitioners pay much attention to this concept. Furthermore, they provided a detailed review of the body of academic research: the ECM domain, its evolution and the characterization of the main topics. In Rickenberg et al., an established ECM research framework is adopted, refined and explained together with its associated elements and working definitions. On this basis, 68 publications were reviewed and classified, and concepts were derived. Prior research was synthesized and findings were integrated in a concept‐centric way. Finally, the authors exposed implications for research and practice, including future trends.

Two works are related to the ECM implementations. Haug [13] included a definition of a process model for ECM implementation in SMEs. This author proposed a new pattern definition for ECM technology development. In [14], Van Rooij explained that legal issues generated by ERP could be similar to those generated by ECM systems. Therefore, it is advised, with appropriate adaptation, to take these issues into account when developing strategies for implementing the ECMs.

ISO 2709:2008 [15] specifies the requirements for a generalized exchange format containing records describing all forms of material capable of bibliographic description as well as other types of records. ISO 15836: 2009 [16] establishes a standard for describing resources across domains known as metadata elements Dublin Core Set. This standard defines the elements that are commonly used in the context of an application profile, limiting their use in accordance with the policies of a particular community and it does not define the implementation details.

ISO 10244:2010 [17] provides businesses with the tools to identify the relevant aspects of the business work processes and document them in a standardized format.

## **3. The THOT project scenario**

The THOT project is an e‐Government project with the objective to implement an ECM system in the Public Administration of Andalusian region of Spain granted for 621,250.00Euros. This chapter is focused on the first phase of the THOT project and explains how the technological and functional status of existing ECM systems has been studied. It is very important to evaluate all existing alternatives in the market in order to align the scope of the organization with its purpose. It is also relevant not to reject the decision because it is difficult to change the chosen system due to the cost, once the development of the solution has started. As the evaluation process concerns, a static evaluation or characterization of an ECM solution is not enough, since new improvements of ECM systems are continuously appearing and one has to compare alternatives dynamically and objectively. In addition, this work considers the different preferences of the elements containing an ECM system in terms of given specific context. This study discusses in detail existing ECM systems in the market and proposes an objective method to compare them within the specific scope and strategic objective of organizations.

Nowadays, the Andalusian Public Administration is driving the need for a change in the following document management systems:


This project aims to cover different disciplines of research and innovation as document management, electronic government, dissemination and Web services integration policies, enabling organizations to provide a common framework for document management. To achieve the aforementioned objectives, the project is being carried out along the following stages and activities:

	- **•** Analysing the current situation
	- **•** Benchmarking existing tools and systems and selecting the most appropriate option
	- **•** Defining new document management functionalities and fitness for certification as a document management system
	- **•** Adjusting Andalusian horizontal documentary series
	- **•** Developing policies to preserve digital documents

generated by ERP could be similar to those generated by ECM systems. Therefore, it is advised, with appropriate adaptation, to take these issues into account when developing strategies for

ISO 2709:2008 [15] specifies the requirements for a generalized exchange format containing records describing all forms of material capable of bibliographic description as well as other types of records. ISO 15836: 2009 [16] establishes a standard for describing resources across domains known as metadata elements Dublin Core Set. This standard defines the elements that are commonly used in the context of an application profile, limiting their use in accordance with the policies of a particular community and it does not define the implementation details. ISO 10244:2010 [17] provides businesses with the tools to identify the relevant aspects of the

The THOT project is an e‐Government project with the objective to implement an ECM system in the Public Administration of Andalusian region of Spain granted for 621,250.00Euros. This chapter is focused on the first phase of the THOT project and explains how the technological and functional status of existing ECM systems has been studied. It is very important to evaluate all existing alternatives in the market in order to align the scope of the organization with its purpose. It is also relevant not to reject the decision because it is difficult to change the chosen system due to the cost, once the development of the solution has started. As the evaluation process concerns, a static evaluation or characterization of an ECM solution is not enough, since new improvements of ECM systems are continuously appearing and one has to compare alternatives dynamically and objectively. In addition, this work considers the different preferences of the elements containing an ECM system in terms of given specific context. This study discusses in detail existing ECM systems in the market and proposes an objective method

to compare them within the specific scope and strategic objective of organizations.

Nowadays, the Andalusian Public Administration is driving the need for a change in the

**•** JUPITER [18], although this situation will change, because this information system it is going

**•** TheERIS‐G3 [19], which is an e‐Government system to manage the electronic procedures to

**•** @rchivA [20], which facilitates the Patrimony Documentation management of Andalusia This project aims to cover different disciplines of research and innovation as document management, electronic government, dissemination and Web services integration policies, enabling organizations to provide a common framework for document management. To achieve the aforementioned objectives, the project is being carried out along the following

business work processes and document them in a standardized format.

implementing the ECMs.

116 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**3. The THOT project scenario**

following document management systems:

stages and activities:

to be replaced by an ERP technology platform

process public records in each public agency

	- **•** Defining the project dissemination plan
	- **•** Defining project dissemination indicators
	- **•** Executing the dissemination plan

In this context, QuEF is proposed as a work‐in‐progress framework to be validated and used for the evaluation of existing ECM systems. This framework offers a suitable methodology to analyse and evaluate ECM solutions dynamically and objectively. In addition, it includes methods to calculate preferences of ECM features and it defines a lifecycle and tool support to enforce the quality continual improvement of an ECM solution.

## **4. The QuEF framework methodology**

QuEF [21] is a work‐in‐progress framework that has been used in this case study (Public Administration of the Regional Government of Andalusia, Spain) in order to validate it. QuEF is a framework to manage quality of entities (products, processes, services or organizations, among others) in any context and domain. In previous work, this framework was used to manage quality in model‐driven Web development methodologies. In addition, this frame‐ work can also be used for consumers to identify the most suitable product or process for them and decide accordingly.

This framework describes templates to define a specific Quality Model for the domain under study. It also offers a method to customize the Quality Model, evaluate it and calculate the preferences of its elements. Besides, the framework includes the definition of a set of phases to enforce continuous quality improvements in the Quality Model. The most important aspect is that quality management is the central quality entity in the Quality Model. Furthermore, a tool support is also implemented in order to promote this solution in real environments. Therefore, users may indeed obtain quality management in an automatic way using QuEF, and hence, computerise the quality management of entities (products, processes, services or organizations, among others) with the final aim to reduce cost and time and improve overall quality. As such, this framework provides


**Table 1** shows the relationship among other standards. It also represents the relation of these standards as well as the best practices and approaches that have been applied to the QuEF framework both to define QuEF itself and to apply it to a specific domain.


**Table 1.** Standards, best practices and approaches related to the QuEF framework.

As **Figure 1** shows, the framework can be used from two points of view: providers', who need to analyse, control, evaluate and improve entities and consumers, who need to compare entities (depending on their context) to decide on the most suitable one for them.

**Figure 1.** Conceptual scheme representing the goals to be achieved with QuEF.

to enforce continuous quality improvements in the Quality Model. The most important aspect is that quality management is the central quality entity in the Quality Model. Furthermore, a tool support is also implemented in order to promote this solution in real environments. Therefore, users may indeed obtain quality management in an automatic way using QuEF, and hence, computerise the quality management of entities (products, processes, services or organizations, among others) with the final aim to reduce cost and time and improve overall

quality. As such, this framework provides

118 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**Standards, best practices and approaches**

**ISO/IEC 20000, ITIL**

**TQM**, **Six Sigma**, **CMMI Planguage, C‐INCAMI or CTQ, among others**

**•** Quality standards and international best practices

**Work application**

instance:

phases.

phases.

analysis.

Operation phase of QuEF.

**Table 1.** Standards, best practices and approaches related to the QuEF framework.

**•** A set of phases to enforce the continuous quality improvement

**•** A tool to support the quality management lifecycle process

**•** Methods for each phase and templates to customize the Quality Model

framework both to define QuEF itself and to apply it to a specific domain.

**•** Multi‐criteria methods to calculate the elements preference value of the Quality Model

**Table 1** shows the relationship among other standards. It also represents the relation of these standards as well as the best practices and approaches that have been applied to the QuEF

on a quality continual improvement of the Quality Model lifecycle.

ISO/IEC 2000 standard and ITIL best practices deal with improving service quality based on a quality continual improvement of the service lifecycle. For instance, ITIL defines Strategy phase, Design phase, Operation phase, Transition phase and quality continual improvement phase. QuEF covers the same idea with a different goal since QuEF framework manages quality‐based

The QuEF framework defines different phases with artifacts, methods and tools for each phase. Most of these approaches could be adapted and applied to some phases of QuEF. They cover the similar aspects between QuEF and quality management Strategy and Operation phase. For

**•** TQM is a management integrative philosophy aims at continuously improving the quality of products and processes. It could be applied to Strategy and QCI phases in QuEF.

**•** Six Sigma is a business process management strategy very similar to TQM working with many established quality‐management tools. Most of them could be used in Strategy phase and

**•** CMMI is a process improvement approach that intends to help organizations improve their performance. Therefore, it could be applied to Strategy and quality continual improvement

**•** C‐INCAMI provides a domain (ontological) model defining all the concepts and relationships needed to design and implement processes. Hence, it could be used in a Strategy and Operation

**•** CTQ could also be applied to the Strategy phase, the Design phase and the Operation phase of QuEF for specifying project context, nonfunctional requirements, measurement, evaluation and

**•** Planguage could be applied to the Strategy phase of QuEF for specifying quality.

It mainly differs from other frameworks in that it focuses on the Quality Model as well as defines a lifecycle where all phases turn around that Quality Model, as shown in **Figure 2**.

**Figure 2.** Quality management based on the Quality Model lifecycle.

Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, so that organizations can choose the most suitable ECM system for their purposes as well as enforce the continuous quality improvement of these systems within the organization. It is based on ITIL v3, but with a difference; it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all quality management efforts on the Quality Model. This means that it incorporates several phases including different objectives and artefacts. The aforementioned phases are


An effective and efficient quality management essentially demands to define the domain under study. Thus, it is important to consider what type of ECM system is concerned. It is not the same to develop an ECM system for a bank, where security stands as a more important quality characteristic, as opposed to the development of an ECM system as presented herein, where usability, functionality and performance are crucial. The purpose of QuEF is not only to assure a clear strategy for quality management but also automatically facilitates a continuous quality improvement by means of generating checklists and documentation, as well as automatic evaluations and plans which control and improve quality and thus, automatically, reduce effort and time.

**Figure 3** shows the specific proposed metamodel for QuEF. There are many definitions in literature trying to clarify what a Quality Model is. In QuEF, it means a set of characteristics and its relationships, which constitutes the base to specify quality requirements and evaluate them. The Quality Model represents its core with quality management revolving around it. This work proposes a Quality Model metamodel consisting of a simplification and adaptation of the ISO/IEC 15939, so that the model implementation can be more flexible and practical. The Quality Model contains Features, Sub‐Feature and Property.


**Figure 3.** Quality metamodel on QuEF.

Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, so that organizations can choose the most suitable ECM system for their purposes as well as enforce the continuous quality improvement of these systems within the organization. It is based on ITIL v3, but with a difference; it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all quality management efforts on the Quality Model. This means that it incorporates several phases including different objectives and

**•** Quality Model Strategy (QMS) phase: This phase is strategic active that focuses on the definition of a quality management strategy. The past, present and future view elements of the Quality Model in the domain under study are essential to achieve an effective and

**•** Quality Model Design (QMD) phase: This is the phase where the Quality Model is finally designed depending on the requirements from the previous phase. It is the model used in

**•** Quality Model Operation (QMO) phase: In this phase, the Quality Model is used to carry out the quality management process. Consequently, the analysis and evaluation manage‐

**•** Quality Model Transition (QMT) phase: This phase describes the processes that execute changes in the Quality Model, in cases where the domain or context changes due to the

**•** Continuous Quality Improvement (CQI) phase: This phase performs all mechanisms to

An effective and efficient quality management essentially demands to define the domain under study. Thus, it is important to consider what type of ECM system is concerned. It is not the same to develop an ECM system for a bank, where security stands as a more important quality characteristic, as opposed to the development of an ECM system as presented herein, where usability, functionality and performance are crucial. The purpose of QuEF is not only to assure a clear strategy for quality management but also automatically facilitates a continuous quality improvement by means of generating checklists and documentation, as well as automatic evaluations and plans which control and improve quality and thus, automatically, reduce effort

**Figure 3** shows the specific proposed metamodel for QuEF. There are many definitions in literature trying to clarify what a Quality Model is. In QuEF, it means a set of characteristics and its relationships, which constitutes the base to specify quality requirements and evaluate them. The Quality Model represents its core with quality management revolving around it. This work proposes a Quality Model metamodel consisting of a simplification and adaptation of the ISO/IEC 15939, so that the model implementation can be more flexible and practical. The

appearance of new trends, but without affecting the Operation phase.

improve quality in all processes in the lifecycle and the Quality Model.

artefacts. The aforementioned phases are

120 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

efficient quality management process.

and time.

the next phase for the quality management performance.

Quality Model contains Features, Sub‐Feature and Property.

ment processes are performed within this phase.

#### **4.1. Case study: applying the QuEF framework to the THOT project scenario**

As previously mentioned, the THOT project is a transfer project carried out in collaboration with the Regional Government of Andalusia (Junta de Andalucía) in Spain. It has two main objectives:


Currently, there are several solutions for this type of systems in the market, although herein, the most appropriate ones have been selected in relation to the scope of the project. Thus, the QuEF framework has been used to obtain the detailed analysis and evaluation of different ECM alternatives. As **Figure 4** shows, two points of view have been identified in the THOT project scenario: ECM system providers who need to analyse, control, evaluate and improve ECM systems and the Public Administration of the Regional Government of Andalusia, which needs to compare ECM systems (depending on the THOT project scenario) to make a decision on the most suitable one to apply.

**Figure 4.** Conceptual scheme representing the application of QuEF framework to the THOT project scenario.

In order to apply the QuEF framework, IWT2 research group (Web Engineering & Early Testing) is developing QuEF‐TS with the aim of automating all processes and artefacts that QuEF defines for each phase. In this particular case, an Enterprise Architect (EA) modelling environment with UML 2.2 is provided.

**Figure 5.** Activity diagram for the tool support use.

Activity diagrams are used to describe the business and operational step‐by‐step workflows of components within the system. **Figure 5** shows the overall flow of control between ECM system providers and Public Administration may be observed (not necessarily registered users) as well as the system administrator (in this case, the IWT2 research group).

Then, the Quality Model lifecycle has been applied to obtain a characterization model of ECM systems. Firstly, the Strategy phase has been established for an effective quality management of ECM systems. In this case, the quality management problem is, in fact, a multiple‐criteria decision‐making or multiple‐criteria decision‐analysis (MCDA) problem, because multiple criteria are used to calculate the preferences regarding the Quality Model elements demand. Besides, it is a multi‐objective optimization problem, as it is important not only to implement the most valuable ECM system but also to reduce cost, risk and uncertainty.

Secondly, the Design phase has been applied to fix the Quality Model that was used to analyse and evaluate quality. The transition phase is also important, since new systems or trends regarding the insertion of new characteristics in the Quality Model can appear in future iterations and these changes must be controlled.

Thirdly, this study will explain how the ECM systems have been analysed and evaluated in the Operation phase. Hence, a tool support has been implemented to automate the generation of all artefacts.

Finally, the CQI phase is shown to clarify how this in conjunction with its phases have been set in order to perform an improved quality cycle encapsulating all phases within the frame‐ work.

#### **4.2. The Quality Model Strategy phase**

to compare ECM systems (depending on the THOT project scenario) to make a decision on the

**Figure 4.** Conceptual scheme representing the application of QuEF framework to the THOT project scenario.

In order to apply the QuEF framework, IWT2 research group (Web Engineering & Early Testing) is developing QuEF‐TS with the aim of automating all processes and artefacts that QuEF defines for each phase. In this particular case, an Enterprise Architect (EA) modelling

Activity diagrams are used to describe the business and operational step‐by‐step workflows of components within the system. **Figure 5** shows the overall flow of control between ECM

most suitable one to apply.

122 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

environment with UML 2.2 is provided.

**Figure 5.** Activity diagram for the tool support use.

In this phase, it is important for the THOT project purposes to explain that the characterization model must be based on concrete solutions for ECM systems and not in work associated with theoretical proposals in the ECM systems context. The main motivation of this decision is the fact that a new ECM system must be developed, which will be based on an existing ECM system. In addition, the high cost this type of projects entails makes it very risky to implement a solution from the ground up. Thus, it is better that all expected requirements and function‐ alities have already been validated in the market.

Then, a characterization model has to be defined in order to analyse and evaluate the systems with QuEF. Carrying out this task demands that the main Features and Sub‐Features of all these systems are known. A systematic literature review (SLR) was used to analyse the current situation. A SLR is a means of identifying, evaluating and interpreting all available documents related to a particular thesis in a specific investigation area.

To perform the SLR, the protocol defined by Kitchenmham [22, 23] was chosen. It is one of the most acknowledged in software engineering. This model establishes the necessity of specifying some research questions (RQs) that will guide the work. For this work, the following RQs were proposed:


The databases considered for this SLR were ACM Digital Library, EiCompendex, IEEE Xplore, ISI Web of Knowledge, Science Direct, SCOPUS, Springer Link and Wiley InterScience Journal Finder.

Once the method was applied, the results showed that the following tools needed to be analysed: Alfresco [24], Documentum [25], Nuexo/Athento [26], IBM FileNet [27] and Open‐ Text [28]. Then, the concept mapping method [29, 30] was executed in order to obtain the characterization model. This method involves all stakeholders in the project. Several meetings were organized with all stakeholders and system providers to discuss requirements in their systems and all this information was used to build the characterization model by the concept mapping method. Concept mapping is a general method that can be used to help any indi‐ vidual or group describe their ideas about some topics in a pictorial form.

In accordance with all these analysed systems and the strategy followed herein, a set of preferences for each element of the Quality Model was defined for adapting it to the project scope.

#### **4.3. The Quality Model Design phase**

The QMD phase in QuEF is understood to encompass all the relevant elements to design the Quality Model. As such, the QMD phase defines the necessary basic characteristics to be analysed using the defined templates. Defining these characteristics, it is possible to asses each solution uniformly.

The characterization scheme is composed of 10 features, which respond to the questions identified in the QMS phase. This priority is contextualized within the needs of this project: to define an innovative solution for document management applied to procurement of services and transport infrastructure projects within the Regional Government of Andalusia, Spain. However, the software allows any potential user utilizing QuEF to set priorities based on his/ her own needs. These basic characteristics are illustrated below:

**FT01: Functional modules**. The results obtained in the QMS phase point out that a valid ECM system must include natively and minimally the following functionalities.

**FT02: User orientation.** Although ECM systems offer standard solutions on its orientation towards the end user, many companies need to use easy and versatile systems because not all their employees have the same user profile to handle computer tools.

**FT03: Functionality to capture, access, retrieve and view documents.** The ability that lets anyone transform the system depending on the organization preferences or user profile.

**FT04: Documental lifecycle.** This feature enables the user to assess the level or degree of support the system offers to the document cycle. The following Sub‐Features of this Feature are described as follows:

**FT05: Workflows.** This Feature assesses whether the tool supports management with business processes.

**FT06: eGovernment.** These Features measure the degree of support offered in the context of access and use digital documents.

**FT07: Interoperability compliance.** A specific section dealing with interoperability has been included: Integration with tools. This Sub‐Feature evaluates whether the ECM system provides mechanisms (e.g. APIs) to integrate with third‐party tools.

**FT08: Security and control.** One of the major objectives of document management solutions is to ensure information security. This is facilitated by controlling access to the system from inside and outside the organization and managing the relevant documents in such a way that they are either archived or destroyed. Consequently, these solutions must provide services that ensure that the information stored is secure. It evaluates whether the system is functional enough to analyze data, or otherwise, whether the system allows using third‐party tools.

**FT09: Architecture.** It evaluates whether the system has an open or closed architecture.

**FT10: Cost.** Cost (both initial and long‐term by maintenance) is one of the most important factors any organization must take into account when choosing an ECM solution.

**FT11: Assistance and RM (Roadmap) support.** This last Feature listed in the latter group includes aspects for the evaluation of the characteristics support, assistance and roadmap provided by the ECM solution.

### **4.4. The Quality Model Transition phase**

**•** RQ3 ‐ What is the most appropriate ECM system that the Andalusian Public Administration, and more specifically, the contracting services for transport and infrastructure construc‐

The databases considered for this SLR were ACM Digital Library, EiCompendex, IEEE Xplore, ISI Web of Knowledge, Science Direct, SCOPUS, Springer Link and Wiley InterScience Journal

Once the method was applied, the results showed that the following tools needed to be analysed: Alfresco [24], Documentum [25], Nuexo/Athento [26], IBM FileNet [27] and Open‐ Text [28]. Then, the concept mapping method [29, 30] was executed in order to obtain the characterization model. This method involves all stakeholders in the project. Several meetings were organized with all stakeholders and system providers to discuss requirements in their systems and all this information was used to build the characterization model by the concept mapping method. Concept mapping is a general method that can be used to help any indi‐

In accordance with all these analysed systems and the strategy followed herein, a set of preferences for each element of the Quality Model was defined for adapting it to the project

The QMD phase in QuEF is understood to encompass all the relevant elements to design the Quality Model. As such, the QMD phase defines the necessary basic characteristics to be analysed using the defined templates. Defining these characteristics, it is possible to asses each

The characterization scheme is composed of 10 features, which respond to the questions identified in the QMS phase. This priority is contextualized within the needs of this project: to define an innovative solution for document management applied to procurement of services and transport infrastructure projects within the Regional Government of Andalusia, Spain. However, the software allows any potential user utilizing QuEF to set priorities based on his/

**FT01: Functional modules**. The results obtained in the QMS phase point out that a valid ECM

**FT02: User orientation.** Although ECM systems offer standard solutions on its orientation towards the end user, many companies need to use easy and versatile systems because not all

**FT03: Functionality to capture, access, retrieve and view documents.** The ability that lets anyone transform the system depending on the organization preferences or user profile.

**•** RQ4 ‐ What areas of improvement are needed for the selected ECM system?

vidual or group describe their ideas about some topics in a pictorial form.

her own needs. These basic characteristics are illustrated below:

system must include natively and minimally the following functionalities.

their employees have the same user profile to handle computer tools.

tions must use?

124 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Finder.

scope.

solution uniformly.

**4.3. The Quality Model Design phase**

The QMT phase provides guidance to undergo changes in the Quality Model without affecting the QMO phase. It helps to know how to handle changes in the Quality Model.

Along this first iteration of the framework, lots of new trends were considered likely to be included in the Quality Model. This study considered in the beginning Alfresco, Documentum, IBM FileNet and OpenText. Then, in a second iteration of the framework, Nuxeo/Athento and KM (SAP 2013) were included. Nevertheless, KM was rejected in the QMS phase because this system did not comply with the outlined project's scope, thus Nuxeo/Athento was finally considered. New Features and Sub‐Features were included in the Quality Model and fixed in the QMD phase.

### **4.5. The Quality Model Operation phase**

This phase provides guidance to analyse, evaluate and plan the CQI of ECM systems. In the THOT project, the decision concerning the suitability of the most appropriate ECM system was taken that was deemed most suitable for it. As a result thereof, the Quality Model and each set of preferences pertaining thereto have been defined. As such, this weighted Quality Model need to be used so as to analyse and evaluate the different systems. Thus, in this phase, the model was implemented to manage quality in ECM systems, which were analysed by means of checklists. These checklists are artefacts that contain all Features, Sub‐Features and Prop‐ erties that have been defined to analyse an entity. Hence, checklists are used in order to know the current state of an ECM system.

In addition, QuEF factory has been developed to automate all tasks in this phase. It allows the user to generate all set of artefacts in each phase of QuEF.

Henceforth, the QuEF factory is being developed as a plug‐in of EA. It means that one has to define the Quality Model in EA and explain in which directory the generation of the QuEF‐O Web application ought to be created (**Figure 6**).


**Figure 6.** QuEF factory.

**Figure 7.** Checklist menu in the QuEF‐O Web application lifecycle.

Then, the QuEF‐O Web application is generated and the user can use all necessary artefacts to analyse and evaluate all ECM systems found. For instance, **Figure 7** shows all checklists that have been generated in terms of the Quality Model that have been defined.


**Figure 8.** Functional modules checklist in the QuEF‐O Web application.

need to be used so as to analyse and evaluate the different systems. Thus, in this phase, the model was implemented to manage quality in ECM systems, which were analysed by means of checklists. These checklists are artefacts that contain all Features, Sub‐Features and Prop‐ erties that have been defined to analyse an entity. Hence, checklists are used in order to know

In addition, QuEF factory has been developed to automate all tasks in this phase. It allows the

Henceforth, the QuEF factory is being developed as a plug‐in of EA. It means that one has to define the Quality Model in EA and explain in which directory the generation of the QuEF‐O

the current state of an ECM system.

**Figure 6.** QuEF factory.

user to generate all set of artefacts in each phase of QuEF.

Web application ought to be created (**Figure 6**).

126 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**Figure 7.** Checklist menu in the QuEF‐O Web application lifecycle.


**Figure 9.** Defined preferences for the sub‐features of Functional modules of the QuEF‐O Web application.

**Figure 8** represents the checklist to analyse the functional modules Feature and Sub‐Features. This set of checklist is used in order to study all ECM systems.

As far as preferences are concerned, these have to beset for each element of the Quality Model. For instance, **Figure 9** shows the preferences defined in the QMS phase for functional modules Feature.

Finally, each ECM system can be evaluated in terms of the project's scope (**Figure 10**), taking into consideration the preferences of each element. Consequently, the user can decide what ECM system is the most suitable for his/her needs. Thus, if new elements have to be included in the evaluation, then new iterations of the framework have to be carried out and all artefacts are generated automatically using the QuEF factory. It is the author's view that this way may lead to cost reduction, effort and time associated with and may even improve quality in the quality management processes.

**Figure 10.** QuEF‐O Web application evaluation interface.

#### **4.6. Quality Continual Improvement phase**

**Figure 8** represents the checklist to analyse the functional modules Feature and Sub‐Features.

As far as preferences are concerned, these have to beset for each element of the Quality Model. For instance, **Figure 9** shows the preferences defined in the QMS phase for functional modules

Finally, each ECM system can be evaluated in terms of the project's scope (**Figure 10**), taking into consideration the preferences of each element. Consequently, the user can decide what ECM system is the most suitable for his/her needs. Thus, if new elements have to be included in the evaluation, then new iterations of the framework have to be carried out and all artefacts are generated automatically using the QuEF factory. It is the author's view that this way may lead to cost reduction, effort and time associated with and may even improve quality in the

This set of checklist is used in order to study all ECM systems.

128 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Feature.

quality management processes.

**Figure 10.** QuEF‐O Web application evaluation interface.

It is very important to consider the continuous quality improvement in a quality management process based on the Quality Model lifecycle. Thus, this objective can be achieved only through a constant monitoring and measurement of all activities and processes involved in quality management. The main goals of this phase are summarized as follows:

	- **◦** Improve the quality of the Quality Model provided
	- **◦** Add new properties and basic characteristics that best fit users' properties and the market
	- **◦** Improve and streamline internal processes of the approaches

Therefore, it is the authors' view that all these practices along the project ought to be considered so as to achieve a continuous quality improvement encompassing all phases of the framework.

## **5. Conclusions and future work**

This chapter shows the results of a research project that aim to implement an ECM system in the Public Administration and has been carried out in a real environment.

The SLR carried out focused on Features and Sub‐Features of ECM solutions that were implemented in existing systems. This is because in the THOT project, the conceptualization of ECM systems is not as relevant as the definition of a set of Features and Sub‐Features to compare existing ECM systems in the market. The conceptualization of ECM systems consti‐ tutes a detailed research to guide providers of these systems. Therefore, getting a common Quality Model including the requirements that all these systems must fulfil in the future is essential.

Nevertheless, this project offers providers and users the opportunity to decide what existing ECM system in the market is the most appropriate for their purpose. This decision is of paramount importance, as it is going to be used to implement all business requirements in Public Administration. In consequence, the high cost prevents changing the development project of the new solution, once started. In other words, the project does not focus on the development of a new ECM system, but on deciding which is the most appropriate one in the market.

This study has proposed an evaluation framework concerning different alternatives of ECM systems, and choosing the most suitable one for the scope and context thereof, with the final aim of warranting continuous quality improvement.

QuEF is a framework to manage quality in any product or process, so it can be applied to any entity. It is based on ITIL v3 with the difference that it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all efforts of quality management on the Quality Model. In addition, the framework also defines protocols and methods to perform each phase, so that all protocols and methods can be systematized. Besides, QuEF‐ Factory is a tool support that can generate a Web application for each phase in terms of a Quality Model (QuEF‐S, QuEF‐D, QuEF‐T, QuEF‐O and QuEF‐QCI). QuEF and its tools improve quality management effectiveness and efficiency, since it clarifies the purposes and objectives of management.

Moreover, a set of tools have been evaluated regarding different aspects. Studying the results, they show that Alfresco System and Nuxeo/Athento should be discarded because of the cost and the functionality offered, respectively. The final user should choose between IBM FileNet and Documentum ECM because both are very similar in terms of cost/value.

## **Acknowledgements**

This research has been partially supported by the NDTQ‐Framework project (TIC‐5789) of Junta de Andalucí́a, by the TEMPROS project of the Spain Ministry (TIN2010‐20057‐C03‐02), by the SoftPLM network, TIN2015‐71938‐REDT, the FEDER Funds and Fujitsu Laboratories of Europe (FLE). We also thank all the staff and researches of the Agency of Public Works of the Regional Government of Andalusia, Spain, for their help, support and professional attitude.

## **Author details**

José González Enríquez\* , Francisco José Domínguez Mayo, Julián Alberto García García, María José Escalona Cuaresma and Manuel Mejías Risoto

\*Address all correspondence to: jose.gonzalez@iwt2.org

Web Engineering and Early Testing (IWT2) Research Group, Computer Languages and System Department, University of Seville, Seville, Spain

## **References**

project of the new solution, once started. In other words, the project does not focus on the development of a new ECM system, but on deciding which is the most appropriate one in the

This study has proposed an evaluation framework concerning different alternatives of ECM systems, and choosing the most suitable one for the scope and context thereof, with the final

QuEF is a framework to manage quality in any product or process, so it can be applied to any entity. It is based on ITIL v3 with the difference that it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all efforts of quality management on the Quality Model. In addition, the framework also defines protocols and methods to perform each phase, so that all protocols and methods can be systematized. Besides, QuEF‐ Factory is a tool support that can generate a Web application for each phase in terms of a Quality Model (QuEF‐S, QuEF‐D, QuEF‐T, QuEF‐O and QuEF‐QCI). QuEF and its tools improve quality management effectiveness and efficiency, since it clarifies the purposes and objectives

Moreover, a set of tools have been evaluated regarding different aspects. Studying the results, they show that Alfresco System and Nuxeo/Athento should be discarded because of the cost and the functionality offered, respectively. The final user should choose between IBM FileNet

This research has been partially supported by the NDTQ‐Framework project (TIC‐5789) of Junta de Andalucí́a, by the TEMPROS project of the Spain Ministry (TIN2010‐20057‐C03‐02), by the SoftPLM network, TIN2015‐71938‐REDT, the FEDER Funds and Fujitsu Laboratories of Europe (FLE). We also thank all the staff and researches of the Agency of Public Works of the Regional Government of Andalusia, Spain, for their help, support and professional attitude.

Web Engineering and Early Testing (IWT2) Research Group, Computer Languages and System

, Francisco José Domínguez Mayo, Julián Alberto García García,

and Documentum ECM because both are very similar in terms of cost/value.

aim of warranting continuous quality improvement.

130 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

market.

of management.

**Acknowledgements**

**Author details**

José González Enríquez\*

María José Escalona Cuaresma and Manuel Mejías Risoto

\*Address all correspondence to: jose.gonzalez@iwt2.org

Department, University of Seville, Seville, Spain


[28] OpenText, available online at http://www.opentext.es/, Accessed 19 July 2016.

[14] Van Rooij, J. C. (2013). Legacy Issues in the Implementation of Enterprise Content

[15] ISO2709:2008, Information and documentation—format for information exchange, http://www.iso.org/iso/iso\_catalogue/catalogue\_tc/catalogue\_detail.htm?csnum‐

[16] ISO15836:2009, Information and documentation—the Dublin Core metadata element set, http://www.iso.org/iso/catalogue\_detail.htm?csnumber=52142, Accessed 4 July

[17] ISO10244:2010, Document management—business process baselining and analysis, http://www.iso.org/iso/home/store/catalogue\_tc/catalogue\_detail.htm?csnum‐

[18] JUPITER Project, Government of Andalucia, http://www.juntadeandalucia.es/ repositorio/ usuario/listado/fichacompleta.jsf?idProyecto=19, Accessed 15 July

[19] ERIS‐G3 Project, Government of Andalucia, http://www.juntadeandalucia.es/ repositorio/ usuario/listado/fichacompleta.jsf?idProyecto=680, Accessed 15 July

[20] Archiva (@rchiva), Government of Andalucia, https://ws024.juntadeandalucia.es/ae/

[21] Domínguez‐Mayo F.J., Escalona M. J., Mejías M., Ross M., & Staples G. 2012. A Quality Management Based on the Quality Model Lifecycle. Computer Standards & Interfaces.

[22] Kitchenham et al. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3. Department of Computer Science, University of

[23] Kitchenham B., Brereton O. P., Budgen D., Turner M., Bailey J., & Linkman S. (2009). Systematic literature reviews in software engineering—A systematic literature review, Information and Software Technology, 51(1), 7–15, ISSN 0950‐5849, http://dx.doi.org/

[25] Documentum, available online at http://www.emc.com/domains/documentum/

[26] Nuxeo/Athento, available online at http://www.athento.com/nuxeo/, Accessed 19 July

[27] IBM FileNet, available online at http://www‐03.ibm.com/software/products/us/en/

[24] Alfresco, available online at http://www.alfresco.com., Accessed 19 July 2016.

admine‐lec/areatecnica/archiva, Accessed 4 July 2013.

Durham, Durham, UK. EBSE‐2007‐01.

10.1016/j.infsof.2008.09.009

index.htm, Accessed 19 July 2016.

filecontmana/, Accessed 19 July 2016.

Management (ECM). International Journal of Information, 3(3), 120–123.

ber=41319, Accessed 4 July 2016.

132 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

ber=45935, Accessed 4 July 2016.

2016.

2016.

2016.

2016.

34(4), 396–412.


#### **Exploring the Relationship of Supply Chain Risk Management to Quality Management Exploring the Relationship of Supply Chain Risk Management to Quality Management**

#### Tyler Florio Tyler Florio

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/65847

#### **Abstract**

This research explores the relationship between an organization's supply chain risk management (SCRM) maturity and quality maturity. SCRM maturity was measured using a survey questionnaire sent to organizations in the USA. Quality maturity was assessed via ISO 9001:2008 certification status as well as through a survey questionnaire of total quality management (TQM) practices for organizations in the USA. The results suggest that ISO 9001:2008 is not related to SCRM maturity, while TQM maturity is related to SCRM maturity. Organizations with more mature TQM programs appear to also have more mature SCRM programs.

**Keywords:** quality management, supply chain management, risk management

## **1. Introduction**

Maintaining the integrity of a supply chain through risk mitigation is crucial to smooth and efficient business operations. However, as supply chains become more global in scope, the potential for risk events occurring increases. For this reason, supply chain risk management has gained substantial interest in recent years among academics. Preliminary research indicates that there are no established standards for certifying risk management capability of organizations in the supply chain. In relation to standards and guidelines, the International Organization for Standardization (ISO) has emerged as a body that seeks to establish and promote best business practices through certifications that organizations can earn. These certifications signal to potential partners that a level of capability has been attained by the organization in a specific area of interest, such as quality management.

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Common certifications related to business include, but are not limited to: ISO 9001, 11000, 14001, and 22000. Most of these involve quality and safety, but none certify for supply chain risk management specifically. The closest standard to achieving this is the ISO 31000 risk management principles and guidelines. However, a certification is not available. Therefore, the rigor of a certification is absent from this standard. In order to be awarded an ISO certification, an organization must submit an application to ISO and undergo a rigorous six‐ stage evaluation process based on various criteria. Other certification bodies do exist that attempt to augment risk management activities in the supply chain. One standard that at least implies an organizational ability to manage risk is Customs‐Trade Partnership Against Terrorism (C‐TPAT). It is a voluntary program that focuses on improving the security of private companies' supply chains with respect to terrorism. However, it addresses only one particular type of risk event related to disruptions produced by actual or potential terrorist threats and does not address an organization's overall risk management maturity. Nonetheless, organiza‐ tional certification by ISO 9001 in particular may be able to signal risk management capabilities simply by virtue of the attention the standards bring to improving process management through the principles of total quality management (TQM). ISO 9001 was revised in 2000 to incorporate the principles of TQM into its certification criteria. Therefore, by extension, TQM maturity, in general, may also provide a signal of supply chain risk management maturity.

A large number of organizations, covering a wide variety of industries, are ISO 9001 certified. Because quality standards and certifications are intended to unify and improve business practices as a whole, the following question arises: are companies that have more mature quality systems, and certified to ISO 9001 in particular, better equipped to manage risk? Researching companies who have quality management programs and how their processes have improved since implementing them may shed light on protecting a company's assets, operations, and its structure from adverse risk events. This research assists in confirming the following statement: A company's use of a TQM system, and particularly through ISO 9001 certification, ensures a high level of risk maturity as compared to that of companies that do not implement a TQM system and/or quality certifications.

In the next section, the literature is reviewed and research questions stated. This is followed by the research methodology, which includes a description of the data. Results are then presented. The findings of this research are subsequently discussed and conclusions drawn. This is followed by a discussion of areas for further study and limitations to the research.

## **2. Literature review**

A study involving supply chain risk starts with a classification of all potential risks. These typically include the following: supply risk, demand risk, process risk, technology risk, logistics risk, information risk, and environment risk [1, 2]. The current research focuses on supply and demand risks because these present supply chain managers with significant challenges due to the severity of the impact and difficulty of effective mitigation. Van Miegham [3] characterizes the loss of a key supplier as having a high effect for aggregate loss severity and a moderate probability of occurrence. Further, Chen et al. [4] report that demand risks have a direct negative effect on supply chain performance. The literature also recommends differing approaches to moderating the occurrence of supply or demand disruptions, such as firm innovativeness, process modularity, and interactive complexity reduction [5–7]. These all relate to the types of activities involved in a quality management strategy that seeks to simplify, standardize, and generally improve products and processes.

As noted, the occurrence of a supply chain risk event can be damaging to any organization no matter where they may be within the supply chain. Technically speaking, risk is defined as the (negative impact to objectives × likelihood of occurrence). Risk management contains four primary steps within its processes. These steps include the following:

**•** Risk identification,

Common certifications related to business include, but are not limited to: ISO 9001, 11000, 14001, and 22000. Most of these involve quality and safety, but none certify for supply chain risk management specifically. The closest standard to achieving this is the ISO 31000 risk management principles and guidelines. However, a certification is not available. Therefore, the rigor of a certification is absent from this standard. In order to be awarded an ISO certification, an organization must submit an application to ISO and undergo a rigorous six‐ stage evaluation process based on various criteria. Other certification bodies do exist that attempt to augment risk management activities in the supply chain. One standard that at least implies an organizational ability to manage risk is Customs‐Trade Partnership Against Terrorism (C‐TPAT). It is a voluntary program that focuses on improving the security of private companies' supply chains with respect to terrorism. However, it addresses only one particular type of risk event related to disruptions produced by actual or potential terrorist threats and does not address an organization's overall risk management maturity. Nonetheless, organiza‐ tional certification by ISO 9001 in particular may be able to signal risk management capabilities simply by virtue of the attention the standards bring to improving process management through the principles of total quality management (TQM). ISO 9001 was revised in 2000 to incorporate the principles of TQM into its certification criteria. Therefore, by extension, TQM maturity, in general, may also provide a signal of supply chain risk management maturity.

A large number of organizations, covering a wide variety of industries, are ISO 9001 certified. Because quality standards and certifications are intended to unify and improve business practices as a whole, the following question arises: are companies that have more mature quality systems, and certified to ISO 9001 in particular, better equipped to manage risk? Researching companies who have quality management programs and how their processes have improved since implementing them may shed light on protecting a company's assets, operations, and its structure from adverse risk events. This research assists in confirming the following statement: A company's use of a TQM system, and particularly through ISO 9001 certification, ensures a high level of risk maturity as compared to that of companies that do

In the next section, the literature is reviewed and research questions stated. This is followed by the research methodology, which includes a description of the data. Results are then presented. The findings of this research are subsequently discussed and conclusions drawn. This is followed by a discussion of areas for further study and limitations to the research.

A study involving supply chain risk starts with a classification of all potential risks. These typically include the following: supply risk, demand risk, process risk, technology risk, logistics risk, information risk, and environment risk [1, 2]. The current research focuses on supply and demand risks because these present supply chain managers with significant challenges due to the severity of the impact and difficulty of effective mitigation. Van Miegham [3] characterizes the loss of a key supplier as having a high effect for aggregate loss severity

not implement a TQM system and/or quality certifications.

136 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**2. Literature review**


Within these four steps are the proper responses to various types of risk. The first possible response is to mitigate risk. Mitigating entails performing an action to reduce the impact or likelihood of various risk events. The second possible response is to avoid risk. Avoiding risk entails completely ceasing the various activities that create said risk. The third response is to transfer risk. Transferring risk involves shifting risk to other operational areas of the supply chain that are better equipped to handle risk events. The fourth and final way to manage risk is to accept risk. Risk may be accepted if its consequences do not outweigh the benefits of surrounding the risk that is created [8]. Three common methods of assessing risk are effective, but not unified in their approach.

The first method is the Delphi method. The Delphi method was originally developed in 500 B.C. by Greek prophets [9]. The prophets would hear various people's complaints, develop a response, and allow the people to formulate a revised complaint. This method was revised by the RAND Corporation in the 1950s and followed the Greek's original method. The RAND Corporation's method consisted of surveys, followed by a response, followed by revised surveys based on initial results. This process gave participants a chance to re‐assess criteria and re‐evaluate based on the responses, which provided greater insight to their issues at hand. The main drawback to this method is it is time consuming, because of the analysis of the initial surveys followed by the revisions of the second round of surveys.

The second popular method is Monte Carlo Simulation. This method was developed in the Monte Carlo casinos to gauge risk brought upon by the various gambler's chances of winning. This method focuses on uncertain risk and is assessed by model construction and analysis through computer simulation. The negative aspect of this method occurs, because it requires extensive mathematical prowess and requires significant amount of education to be proficient.

The third method is decision tree analysis. Decision tree analysis utilizes graphical methods to draw correlations to common risks. This method is effective, because it creates a visual image of how various risks are linked, but suffers because of its simplicity [10].

Normally, since avoiding risk is so vital, there should be a series of guidelines in place to facilitate conducting supply chain risk management (SCRM) while supporting total quality management (TQM). Performed literature review indicates that there is no unified series of standards of avoiding risk. Even though most companies have their own emergency prepar‐ edness plans in place, a majority of company executives do not review or approve them and only 42% conduct emergency preparedness practices on a regular basis [11]. Companies must learn how to handle diverse amounts of risk such as natural disasters, political unrest, or acts of terrorism. Prior to September 11, 2001, preparedness levels addressing terrorism did not exist.

The World Economic Forum (WEF) has reported that although supply chain risk is an important issue, it is widely mismanaged [12]. Consistent mismanagement of risk across multiple industries might have a ripple effect on global risk which tends to amplify the disruptive impacts of a local risk event with resulting impacts far beyond the corporate sector. The believed cause is companies' inability to detect these ripples (such as the effects of catastrophes and pandemics on the other side of the world) before they become waves that disrupt their supply chains [13]. Many of these preparedness plans have been found to be inefficient because of lack of communication and collaboration, such as Alabama and Louisi‐ ana's responses to Hurricane Katrina [14].

Along with natural disasters and man‐made risk, there is organizational and network risk. Organizational risks include inventory risk, process/operational risk, quality risk, and management risk, while network risks result from interactions between organizations within the supply chain. Agility, flexibility, contingency planning, and preparedness are preferred generic strategies for managing such risks in general [15]. It has been found that larger companies in the private sector are better equipped to individually handle disaster than smaller companies in the public sector [16–18]. These small companies simply do not have the resources to develop a proactive approach, so instead they take a more defensive (reactive) approach constituting risk elimination aspects [19]. Performed research indicates a gap between the risk management policies that large companies are capable of implementing and what their smaller counterparts are able to produce. Many have tried to develop supply chain risk management policies linking risk identification, risk assessment, and risk mitigation to risk performance [20]. Academics along with industry leaders see a need to replace traditional and varying risk management techniques for ones that are better designed to handle extreme complexities, unpredictable events, and threats. They have sought to discover a link between vulnerable factors and controllable factors. The Supply Chain Resilience Assessment and Management (SCRAM) framework was developed, but only served to suit the needs of a select few companies [21]. Companies must also have a framework in place to address general risk between inter‐organizational partnering. Once two or more companies become partners, they assume the other's risk in some or a large capacity, as previously mentioned [22]. According to Zhao et al. [23], "…*supplier, internal, and customer integration are the most important drivers for schedule attainment, competitive performance, and customer satisfaction, respectively"* (p. 115). They find that supply chain risks are negatively related to supply chain integration. Although global integration is crucial and necessary to be competitive, it inherently carries an increased amount of risk.

Performed literature review shows that multiple organizations and companies have tried to institute best practices for mitigating risk. However, there is not one general set standard for how to approach the subject of avoiding or correcting risk in the supply chain. The reason for this variety in practices is possibly because of the variety of industries coupled with the varying ways in which risk can present itself. The food industry will not have the same problems as a metal manufacturing industry. Even within the same industry, companies might not have the same problems. A seafood distributor will have different issues to assess than a fruit producer. These varying sources and types of risk have made it difficult to create guidelines in order to steer companies in the right direction to manage risk in their supply chain. Also, since there is a lack of general guidelines concerning risk management, every industry (even every compa‐ ny) has taken it upon themselves to develop their own guidelines.

The International Standard for Organization (ISO) seeks to remedy the varying levels of supply chain risk preparedness and quality by allowing companies to become universally certified in various ISO certifications to promote a unity of practice.

The steps for ISO certification are as follows:

Normally, since avoiding risk is so vital, there should be a series of guidelines in place to facilitate conducting supply chain risk management (SCRM) while supporting total quality management (TQM). Performed literature review indicates that there is no unified series of standards of avoiding risk. Even though most companies have their own emergency prepar‐ edness plans in place, a majority of company executives do not review or approve them and only 42% conduct emergency preparedness practices on a regular basis [11]. Companies must learn how to handle diverse amounts of risk such as natural disasters, political unrest, or acts of terrorism. Prior to September 11, 2001, preparedness levels addressing terrorism did not

The World Economic Forum (WEF) has reported that although supply chain risk is an important issue, it is widely mismanaged [12]. Consistent mismanagement of risk across multiple industries might have a ripple effect on global risk which tends to amplify the disruptive impacts of a local risk event with resulting impacts far beyond the corporate sector. The believed cause is companies' inability to detect these ripples (such as the effects of catastrophes and pandemics on the other side of the world) before they become waves that disrupt their supply chains [13]. Many of these preparedness plans have been found to be inefficient because of lack of communication and collaboration, such as Alabama and Louisi‐

Along with natural disasters and man‐made risk, there is organizational and network risk. Organizational risks include inventory risk, process/operational risk, quality risk, and management risk, while network risks result from interactions between organizations within the supply chain. Agility, flexibility, contingency planning, and preparedness are preferred generic strategies for managing such risks in general [15]. It has been found that larger companies in the private sector are better equipped to individually handle disaster than smaller companies in the public sector [16–18]. These small companies simply do not have the resources to develop a proactive approach, so instead they take a more defensive (reactive) approach constituting risk elimination aspects [19]. Performed research indicates a gap between the risk management policies that large companies are capable of implementing and what their smaller counterparts are able to produce. Many have tried to develop supply chain risk management policies linking risk identification, risk assessment, and risk mitigation to risk performance [20]. Academics along with industry leaders see a need to replace traditional and varying risk management techniques for ones that are better designed to handle extreme complexities, unpredictable events, and threats. They have sought to discover a link between vulnerable factors and controllable factors. The Supply Chain Resilience Assessment and Management (SCRAM) framework was developed, but only served to suit the needs of a select few companies [21]. Companies must also have a framework in place to address general risk between inter‐organizational partnering. Once two or more companies become partners, they assume the other's risk in some or a large capacity, as previously mentioned [22]. According to Zhao et al. [23], "…*supplier, internal, and customer integration are the most important drivers for schedule attainment, competitive performance, and customer satisfaction, respectively"* (p. 115). They find that supply chain risks are negatively related to supply chain integration. Although global

exist.

ana's responses to Hurricane Katrina [14].

138 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Stage 1: Proposal stage—development of initial proposal of operational standards

Stage 2: Preparatory stage—preparing proposal for submission

Stage 3: Committee stage—committee review of proposal

Stage 4: Enquiry stage—questioning into company operations post‐review

Stage 5: Approval stage—approval for ISO certification

Stage 6: Publication stage—ISO certification publicized and legitimized

Certifications, such as ISO 9001, and guidelines, such as ISO 31000, are best suited to prepare companies to handle risk (www.iso9001.com, 1) (www.iso.org, 1). Specifically, ISO 31000 (containing ISO Guide 73:2009 and ISO/IEC 31010:2009) seeks to establish guidelines for risk. ISO 31000 contains within it risk management principles, a framework for risk management, and a process for managing risk. It purports to be applicable to any organization regardless of size or sector and claims to increase the likelihood of improving the identification of oppor‐ tunities and threats, thereby allowing an organization to more effectively allocate and use resources for risk treatment. ISO 31000 does not allow for certification but can aid with internal and external audits. These guidelines allow preparation for strategic, operational, and management risk and provide insight into the philosophy and practices that an organization might adopt in building an effective risk management system.

The ISO 9001 standards are also closely related to the approach used in quality improvement systems. In fact, the literature reports that there is a relationship between TQM and ISO 9001. TQM principles were incorporated into the ISO 9001 standard in the year 2000, so a significant relationship is likely to exist [24]. Studies have since shown a relationship between ISO 9001 and TQM. Psomas et al. [25] found that ISO 9001 certified companies achieve significant quality improvement through the implementation of the core process management practices charac‐ teristic of TQM. Sampaio et al. [26] found that most researchers in this area reason that ISO 9001 and TQM are very similar and should both be implemented in the organization together [27–29]. Others suggest that the structure of ISO 9001 can actually aid in the implementation of TQM practices [30]. Therefore, one would expect TQM principles and practices to be present in organizations that have achieved ISO 9001 certification. A few supply chain risk and quality studies do exist. Chapman et al. [31] studied delivery lead time variability and quality management. Tse et al. [32] explored quality and safety problems in the supply chain and introduced a supply chain risk management framework to reduce quality risk. Therefore, the relationship of supply chain risk management to quality management practices has not been widely studied. Based on this review of the literature, the following statements will be assessed empirically:


## **3. Data and research methodology**

Data for this study came from a list of approximately 3000 United States–based organizations, compiled from the 2014 IAAR Directory, ISM‐CV, and CSCMP member lists. These organiza‐ tions cover varying locations throughout the United States spanning industries such as medical, manufacturing, government, etc. The approach used in this study is a survey questionnaire, conducted via e‐mail, that includes demographic questions and questions specific to quality and risk management practices to gauge the relationship of quality man‐ agement practices and risk management practices of organizations. The answers to the survey questions are then used to compare the level of risk management between organizations who more effectively use quality improvement practices and those who do not.

The survey was distributed to potential respondents using Qualtrics survey software. The survey components contained questions measuring the respondent's position within the organization, their familiarity with the organization's quality management and risk manage‐ ment practices, the organization's size, industry and position within the supply chain, where and how the organization manages risk, the company's quality management practices, and their ISO 9001 certification status. The answers to most of these questionnaire statements were then ranked with answers on a 1–5 scale (1 being the worst and 5 being the best). These responses were analyzed using SPSS software. Selected items from the survey questionnaire are provided in the Appendix. These statements include the number of responses in each category, as appropriate.

The measurement of quality management practice within organizations was taken from the work of Dellana and Kros [33]. They developed a set of 13 statements that cover the main quality issues in an organization that could be found at any point in the supply chain and also in any type of industry. In their study, these 13 statements were split into two distinct con‐ structs. Based on data from 565 respondents, 8 of the statements were assigned to the construct of internal‐downstream quality (i.e., customer‐related), while the remaining 5 statements were grouped under the construct of external‐upstream quality (i.e., supplier‐related). In their investigation, the internal‐downstream construct was found to be associated with industry class and a measure of supply chain position, while the external‐upstream construct was not. Therefore, in the current research, the measure of quality maturity is limited to the eight statements from Dellana's and Kros's previous work that made up the internal‐downstream construct (ref. question 9 of the Appendix). The measurement of risk management practice in organizations was more challenging. No generally accepted measure of risk management maturity was found in the literature. Therefore, a set of statements was inspired and derived from a number of sources, including the ISO 31000 guidelines [34–42]. This resulted in the 15 statements listed in the questionnaire for question 7 of the Appendix.

## **4. Results**

and TQM. Psomas et al. [25] found that ISO 9001 certified companies achieve significant quality improvement through the implementation of the core process management practices charac‐ teristic of TQM. Sampaio et al. [26] found that most researchers in this area reason that ISO 9001 and TQM are very similar and should both be implemented in the organization together [27–29]. Others suggest that the structure of ISO 9001 can actually aid in the implementation of TQM practices [30]. Therefore, one would expect TQM principles and practices to be present in organizations that have achieved ISO 9001 certification. A few supply chain risk and quality studies do exist. Chapman et al. [31] studied delivery lead time variability and quality management. Tse et al. [32] explored quality and safety problems in the supply chain and introduced a supply chain risk management framework to reduce quality risk. Therefore, the relationship of supply chain risk management to quality management practices has not been widely studied. Based on this review of the literature, the following statements will be assessed

**1.** Organizations that are ISO 9001 certified have more mature risk management systems

**2.** Organizations that have more mature risk management systems also demonstrate more

Data for this study came from a list of approximately 3000 United States–based organizations, compiled from the 2014 IAAR Directory, ISM‐CV, and CSCMP member lists. These organiza‐ tions cover varying locations throughout the United States spanning industries such as medical, manufacturing, government, etc. The approach used in this study is a survey questionnaire, conducted via e‐mail, that includes demographic questions and questions specific to quality and risk management practices to gauge the relationship of quality man‐ agement practices and risk management practices of organizations. The answers to the survey questions are then used to compare the level of risk management between organizations who

The survey was distributed to potential respondents using Qualtrics survey software. The survey components contained questions measuring the respondent's position within the organization, their familiarity with the organization's quality management and risk manage‐ ment practices, the organization's size, industry and position within the supply chain, where and how the organization manages risk, the company's quality management practices, and their ISO 9001 certification status. The answers to most of these questionnaire statements were then ranked with answers on a 1–5 scale (1 being the worst and 5 being the best). These responses were analyzed using SPSS software. Selected items from the survey questionnaire are provided in the Appendix. These statements include the number of responses in each

The measurement of quality management practice within organizations was taken from the work of Dellana and Kros [33]. They developed a set of 13 statements that cover the main

more effectively use quality improvement practices and those who do not.

than organizations that are not ISO 9001 certified.

mature quality management systems.

140 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**3. Data and research methodology**

category, as appropriate.

empirically:

The results of the survey based on SPSS statistical software analysis are as follows. A total of 500 of the approximately 3000 potential respondents reached an active e‐mail account. Of these 500, responses to the questionnaire were received from 40 individuals. Of these 40 responses, 18 respondents indicated their organization as ISO 9001 certified, while 20 responded that their organization was not ISO 9001 certified (two did not respond to the ISO certification question and therefore these two questionnaires were unusable). Analysis of nonresponse bias was performed by comparing the mean score for TQM of early versus late responders. There was no significant difference in the mean score between the two groups (t = 1.40, p = 0.18). However, given the small sample size, a comparison of means is by no means conclusive. Therefore, this study and its findings are considered preliminary and exploratory in nature. It should also be noted that this survey was conducted prior to the issuing of the ISO 9001:2015 standards, which incorporate a focus on risk management. The standards in force at the time of this research were ISO 9001:2008.

A general consideration of the survey responses related to demographics of the individual and organization follows. The job title and level of the respondents was generally varied. However, most of the respondents were at the manager level and above (79%). Most were somewhat to very familiar with their organization's supply chain risk management process (78%), with 38% reporting being very familiar. The overwhelming majority were somewhat to very familiar with their organization's quality management practices (94%), with 53% reporting being very familiar. The organizational size, based on number of employees, also varied quite a lot. Grouping into three categories of small, medium, and large yields relative percentages of 42, 29, and 29%, respectively. About 40% of respondent organizations have a department dedi‐ cated to supply chain risk management, while about 32% have each department responsible for assessing supply chain risk related to their particular function. Approximately, 35% are involved in the manufacturing part of the supply chain, while about 38% are involved in distribution. The rest appear to be various services that relate to differing parts of the supply chain. The organizations fall mostly into the general industry categories related to manufac‐ turing (34%) and services (34%). This is followed by transportation/distribution at about 21%. Finally, 47% of respondents reported their organizations are ISO 9001 certified while 53% reported not having ISO 9001 certification.

A factor analysis using principal component analysis was performed on the eight TQM‐related questions to assess whether or not they were consistent with prior research. However, it should be noted that, for the purpose of factor analysis, the present sample did not meet the require‐ ment that minimum sample size be at least 10 times the number of variables per Nunnally [43] (*n* = 40, <10 × 8 = 80). That said, these 8 statements were already shown in Dellana and Kros's previous study to have met the requirements for factor analysis.

It was found that the eight TQM‐related statements all loaded strongly on a single factor. Metrics related to factor analysis were run. The KMO measure of sampling adequacy (index  = 0.863) and Bartlett's test of sphericity (p = 0.0000) were found to be acceptable. Therefore, the results presented herein were very consistent with prior work, which gave support for this research. The model met underlying assumptions (except sample size issues). The reliability of all eight questions together also was very good (Cronbach alpha reliability = 0.93). The TQM maturity variable value used in SPSS analysis was the average score for the eight statements out of a possible score of 5.0. Therefore, a higher average score indicates greater TQM maturity.


**Table 1.** Results of factor analysis for supply chain risk management (SCRM) measurement.

Because the result was so positive for the TQM measure, factor analysis was also conducted on the supply chain risk management (SCRM) statements using principal components analysis in SPSS. Once again, the sample size requirement of Nunnally was not met in this case, so the analysis is preliminary. The KMO measure of sampling adequacy (index = 0.783) and Bartlett's test of sphericity (p = 0.0000) were found to be acceptable. Two factors were extracted using a varimax rotation and Kaiser normalization. The reliability of the statements in the two SCRM factors held very well with Cronbach's alpha measures of 0.88 and 0.86 for Factor 1 and Factor 2, respectively. The two SCRM factor sets are described in **Table 1** and are synonymous with the survey statement designations. Factor 1 seems to describe more so "What" is managed during risk assessment, while Factor 2 seems to describe "How" this risk is assessed and corrected. An overall measure of SCRM combines these two sets of statements for a total of nine statements. In all cases of the risk maturity variables (i.e., SCRM, SCRM1 and SCRM2) the value used in SPSS analysis was the average score for the related statements out of a possible score of 5.0. Therefore, a higher average score indicates greater SCRM maturity.

#### **4.1. Research question 1**

distribution. The rest appear to be various services that relate to differing parts of the supply chain. The organizations fall mostly into the general industry categories related to manufac‐ turing (34%) and services (34%). This is followed by transportation/distribution at about 21%. Finally, 47% of respondents reported their organizations are ISO 9001 certified while 53%

A factor analysis using principal component analysis was performed on the eight TQM‐related questions to assess whether or not they were consistent with prior research. However, it should be noted that, for the purpose of factor analysis, the present sample did not meet the require‐ ment that minimum sample size be at least 10 times the number of variables per Nunnally [43] (*n* = 40, <10 × 8 = 80). That said, these 8 statements were already shown in Dellana and Kros's

It was found that the eight TQM‐related statements all loaded strongly on a single factor. Metrics related to factor analysis were run. The KMO measure of sampling adequacy (index  = 0.863) and Bartlett's test of sphericity (p = 0.0000) were found to be acceptable. Therefore, the results presented herein were very consistent with prior work, which gave support for this research. The model met underlying assumptions (except sample size issues). The reliability of all eight questions together also was very good (Cronbach alpha reliability = 0.93). The TQM maturity variable value used in SPSS analysis was the average score for the eight statements out of a possible score of 5.0. Therefore, a higher average score indicates greater TQM maturity.

> **SCRM Factor 2 statements** (Questionnaire statements b, d, e, f)

annually

b. Risk management is an ad hoc process for us that

d. Our risk management assessment is based on

e. Our risk management assessment occurs at least

f. Risk assessment is a quantitative process for us

occurs informally on an as‐needed basis

probability analysis of relevant risks

reported not having ISO 9001 certification.

142 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**SCRM Factor 1 statements**

education, and treatment

potential supply chain risks

potential supply chain risks

management process (e.g., ISO 31000)

organization

(Questionnaire statements i, k, l, m, n)

i. We follow the four step process of risk identification, analysis,

k. We prioritize risk events based on severity of impact to our

l. We involve our suppliers in identification and mitigation of

m. We encourage our suppliers to use a structured risk

n. We work with our customers to identify and mitigate

**Table 1.** Results of factor analysis for supply chain risk management (SCRM) measurement.

Because the result was so positive for the TQM measure, factor analysis was also conducted on the supply chain risk management (SCRM) statements using principal components analysis in SPSS. Once again, the sample size requirement of Nunnally was not met in this case, so the analysis is preliminary. The KMO measure of sampling adequacy (index = 0.783) and Bartlett's test of sphericity (p = 0.0000) were found to be acceptable. Two factors were extracted using a varimax rotation and Kaiser normalization. The reliability of the statements in the two SCRM

previous study to have met the requirements for factor analysis.

The first research question explores whether organizations that are ISO 9001 certified have more mature risk management systems than organizations that are not ISO 9001 certified.

The relationship of ISO certification with SCRM was measured with the dependent variable entered as the respective risk management variables (i.e., SCRM1, SCRM2, and SCRM) using a univariate linear regression in SPSS with ISO 9001 certification as an independent variable. In all cases, the number of employees, supply chain position, and TQM score were also entered as control variables. Because of low frequencies by category, some data consolidation was performed. The number of employees was collapsed into small, medium, and large groups. Supply chain position was generally sorted into manufacturing and distribution and other.

The results of the regression analysis are given in **Table 2**. Backward removal of variables in SPSS resulted in the ISO 9001 reaching a significance of at best p = 0.064. This was for the case of the SCRM overall. SCRM2 was second best with a significance of p = 0.079. Neither of these make the generally accepted threshold of 0.05 significance level and were, therefore, not included in the final SPSS models. The models were also run with all control variables excluded to determine whether this would make a difference to the result. The outcome was similar to the model that included control variables, with SCRM1 at a significance of p = 0.806, SCRM2 at a significance of p = 0.061, and SCRM at a significance p = 0.248. Therefore, the evidence is weak for a conclusion that ISO 9001 certified organizations have more mature risk management systems than non‐ISO 9001 certified organizations.

A breakdown by industry type had been a goal of this research, but the sample size was too small to accommodate a rigorous statistical analysis. However, it is worth noting that the split of ISO and non‐ISO certified organizations was quite different between organizations classified as manufacturing (9 certified, 4 noncertified) and those classified as service (2 certified and 11 noncertified). Distribution‐related organizations were more evenly split (five certified and three noncertified). **Table 3** shows a breakdown of the scores for SCRM and TQM by major industry type. The scores for SCRM seemed to favor the service organization over manufac‐ turing, especially for SCRM2. Although ISO 9001 does not clearly differentiate regarding SCRM maturity, there is at least the implication in this research that industry type may be a differentiator. A similar analysis could be conducted by supply chain position. It might be expected that organizations further downstream in the supply chain would exhibit greater SCRM maturity than those upstream given the lengthening of the supply chain heading downstream, which increases the chance of risk event occurrence that disrupts the supply chain.


**Table 2.** Linear regression analysis results for SCRM variables versus ISO 9001.


**Table 3.** Mean SCRM and TQM scores by general industry type.

#### **4.2. Research question 2**

The second research question explores whether organizations that have more mature supply chain risk management systems also demonstrate more mature quality management systems. The simple linear regression analysis of **Table 2** incorporates the TQM score as an independent variable in the analysis of relationships with SCRM variables. TQM was found to be strongly significantly related to SCRM1 (p < 0.000) and SCRM overall (p = 0.002). Both of these met the generally accepted threshold of 0.05 significance level and were, therefore, included in the final SPSS models. SCRM Factor 1 statements tend to describe the actual risk management process details, outlining the process steps and who is involved. SCRM Factor 2 statements are more general and have more to do with the timing and nature of the process. The models were also run for SCRM variables with TQM as the only independent variable in each case to determine whether this would make a difference to the result. The outcome was similar to the model that included control variables with SCRM1 at a significance of p < 0.000, SCRM2 at a significance of p = 0.248, and SCRM at a significance p = 0.003. This suggests that the presence of a mature TQM program may signal a more mature supply chain risk management system based on the actual process scope and steps.

Analysis was also run on the correlation between TQM and SCRM1 specifically to see where there were strong relationships between specific quality management statements and supply chain risk management maturity statements. SCRM1 statements i, k, and l were all positively correlated to TQM statements a, b, c, d, e, g, and h (see **Table 1** for the specific questions). SCRM1 statement m was positively correlated to TQM statements a, b, c, g, and h. SCRM1 statement n, b, g, h was positively correlated to TQM statements b, g, and h. Therefore, statement f, "Quality metrics or standards are kept," was not correlated to any SCRM1 statements. This analysis suggests that TQM is strongly related to most of the SCRM1 state‐ ments, with only a few weak relationships. Statement 6 stands out as unrelated to SCRM1, while statement n of SCRM1, "We work with our customers to identify and mitigate potential supply chain risks", stands out as only weakly related to TQM.

## **5. Conclusions**

downstream, which increases the chance of risk event occurrence that disrupts the supply

0.209 0.630 0.117 −0.070

0.117 0.280 0.310 −0.059

0.154 0.547 0.304 −0.119

**Score category Manufacturing (n = 13) Service (n = 13) Distribution (n = 8)**

The second research question explores whether organizations that have more mature supply chain risk management systems also demonstrate more mature quality management systems. The simple linear regression analysis of **Table 2** incorporates the TQM score as an independent variable in the analysis of relationships with SCRM variables. TQM was found to be strongly significantly related to SCRM1 (p < 0.000) and SCRM overall (p = 0.002). Both of these met the

SCRM 3.13 **3.71** 3.56 SCRM1 3.32 **3.60** 3.58 SCRM2 2.84 **3.84** 3.53 TQM 4.02 4.05 **4.32**

**Table 2.** Linear regression analysis results for SCRM variables versus ISO 9001.

**Table 3.** Mean SCRM and TQM scores by general industry type.

1.519 4.516 0.743 −0.467

0.625 1.653 1.814 −0.324

0.903 3.469 1.928 −0.714 0.139 **0.000** 0.464 0.644

0.537 0.109 0.079 0.749

0.374 **0.002** 0.064 0.481

**Variable Beta t Significance**

144 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

chain.

SCRM1

SCRM2

SCRM

**•** SC position **•** TQM **•** ISO 9001

**•** SC position **• TQM •** ISO 9001

**•** SC position **• TQM •** ISO 9001

**•** Number employees

**•** Number employees

**•** Number employees

**4.2. Research question 2**

These results suggest that ISO 9001 is not strongly related to supply chain risk management maturity. There was not a strong relationship between ISO certification and company's preparedness level of SCRM and TQM. However, TQM is clearly related to SCRM1, no matter the variables that are included. The more mature a company's TQM practices, the more mature the company is in terms of SCRM as identified by the SCRM Factor 1 (which describes the actual risk management process details, outlining the process steps and who is involved). Most of the statements comprising this factor were positively correlated to the TQM statements (denoting total quality management practices). In particular, these included the following SCRM1 statements:


The statement related to customers (i.e., "We work with our customers to identify and mitigate potential supply chain risks") was only weakly associated to TQM.

The lack of significance between ISO9001:2008 and the two factors of SCRM, based on sur‐ vey results and analysis, suggests that the standards for ISO 9001:2008 and 31000 do not pro‐ vide a significant advantage to risk assessment and management.

Though they provide positive frameworks for overall company structure and project devel‐ opment, ISO 9001:2008 was not equipped to provide a framework for risk management that eclipses alternative methods. It should be noted that ISO 31000 is a new addition and has not yet been fully developed. Before the implementation of ISO 31000, risk was not explicitly addressed under ISO standards.

## **6. Limitations and further research**

The number of respondents in this study was relatively small and necessarily reduces the power of the statistical tests in this study. The low response rate also introduces the potential for nonresponse bias. Therefore, the conclusions should be treated with caution. However, the results suggest that a higher response rate would indicate a similar pattern of results based on prior studies, in particular related to the TQM maturity metric.

To further aid such a study, case studies may be performed to monitor the daily processes of companies who are and are not ISO certified. This would provide more of an in depth analy‐ sis of the effects of ISO certification on risk management. E‐mail provided certain limitations to accessibility, because of company filtering software and lack of personal interaction. Joint participation with the ISO organization into the effectiveness of their certifications would al‐ so lead to a better understanding and greater ease of access.

However, the results of this initial survey and analysis suggest that ISO does not play a sig‐ nificant role in risk assessment and correction. Though, the more mature a company is able to be in terms of their total quality management procedures, the better equipped it will be to handle supply chain risk.

Further study should be conducted to seek a larger sample size in order to assess the relia‐ bility of the results in this study. It would also help shed light on differences by industry type and supply chain position. However, this will necessarily involve the newer standards for ISO 9001:2015. Therefore, it is unlikely that this study can be replicated. It would, none‐ theless, still be of interest to assess the degree to which the new standards have impacted on organizational effectiveness in managing risk in the supply chain. Since the new standards now focus attention specifically on risk management, it should signal to customers that ISO 9001:2015 certification is a good supplier prequalifier when it comes to supply chain risk management and not just quality. Or at least that would be the expectation. Further study is needed to confirm this.

**Appendix: Selected survey questions** (number responding given in parentheses)

	- **•** CEO/President (7)
	- **•** Vice President (8)
	- **•** Director (4)

The statement related to customers (i.e., "We work with our customers to identify and mitigate

The lack of significance between ISO9001:2008 and the two factors of SCRM, based on sur‐ vey results and analysis, suggests that the standards for ISO 9001:2008 and 31000 do not pro‐

Though they provide positive frameworks for overall company structure and project devel‐ opment, ISO 9001:2008 was not equipped to provide a framework for risk management that eclipses alternative methods. It should be noted that ISO 31000 is a new addition and has not yet been fully developed. Before the implementation of ISO 31000, risk was not explicitly

The number of respondents in this study was relatively small and necessarily reduces the power of the statistical tests in this study. The low response rate also introduces the potential for nonresponse bias. Therefore, the conclusions should be treated with caution. However, the results suggest that a higher response rate would indicate a similar pattern of results based on

To further aid such a study, case studies may be performed to monitor the daily processes of companies who are and are not ISO certified. This would provide more of an in depth analy‐ sis of the effects of ISO certification on risk management. E‐mail provided certain limitations to accessibility, because of company filtering software and lack of personal interaction. Joint participation with the ISO organization into the effectiveness of their certifications would al‐

However, the results of this initial survey and analysis suggest that ISO does not play a sig‐ nificant role in risk assessment and correction. Though, the more mature a company is able to be in terms of their total quality management procedures, the better equipped it will be to

Further study should be conducted to seek a larger sample size in order to assess the relia‐ bility of the results in this study. It would also help shed light on differences by industry type and supply chain position. However, this will necessarily involve the newer standards for ISO 9001:2015. Therefore, it is unlikely that this study can be replicated. It would, none‐ theless, still be of interest to assess the degree to which the new standards have impacted on organizational effectiveness in managing risk in the supply chain. Since the new standards now focus attention specifically on risk management, it should signal to customers that ISO 9001:2015 certification is a good supplier prequalifier when it comes to supply chain risk management and not just quality. Or at least that would be the expectation. Further study is

potential supply chain risks") was only weakly associated to TQM.

146 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

vide a significant advantage to risk assessment and management.

prior studies, in particular related to the TQM maturity metric.

so lead to a better understanding and greater ease of access.

addressed under ISO standards.

handle supply chain risk.

needed to confirm this.

**6. Limitations and further research**

	- **•** 20 or fewer (5)
	- **•** 21 to 100 (11)
	- **•** 101 to 500 (6)
	- **•** 501 to 5000 (5)
	- **•** 5001 to 10,000 (3)
	- **•** More than 10,000 (8)
	- **•** Not sure
	- **a.** Aerospace manufacturing (2)
	- **b.** Automotive manufacturing (1)
	- **c.** Chemicals/plastics manufacturing (2)
	- **d.** Computer equipment manufacturing (0)
	- **e.** Electronics/electrical manufacturing (1)
	- **f.** Fabricated metal products manufacturing (2)
	- **g.** Food products manufacturing (0)
	- **h.** Government/public administration (1)
	- **i.** Machinery manufacturing (Industrial/Commercial) (4)
	- **j.** Pharmaceuticals manufacturing (1)
	- **k.** Retail trade/merchandising (0)
	- **•** We outsource our supply chain risk analysis to a consulting firm. (1)
	- **•** We have a dedicated Risk Management Department (i.e., an internal consultant) that performs supply chain risk analysis for the organization. (9)
	- **•** Each department is responsible for assessing supply chain risk related to their particular function. (12)
	- **•** We have a specific department (e.g., procurement) that is charged with managing supply chain risk. (Please specify department:) (6)
	- **•** Other. Specify: (9)
	- **•** Raw/basic material manufacture (1)
	- **•** Component/sub‐assembly manufacture (1)
	- **•** Final product manufacture (11)
	- **•** Warehousing/distribution (14)
	- **•** Retail (0)
	- **•** Other. Specify: (10)
	- **•** Very unfamiliar (7)
	- **•** Unfamiliar (1)
	- **•** Somewhat familiar (7)
	- **•** Familiar (8)
	- **•** Very familiar (14)

**7.** The following statements relate to your organization's supply chain risk management system. Please indicate the degree to which you agree or disagree with each statement in relation to your organization.

(1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, 5 = Strongly agree)


\*Coded in reverse in the analysis.

	- **•** Very unfamiliar (1)
	- **•** Unfamiliar (1)

**l.** Services: Educational services (1)

148 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**p.** Warehousing/distribution (7)

**n.** Transportation (1)

**q.** Wholesale trade (2)

**r.** Other. Specify: (1)

function. (12)

**•** Other. Specify: (9)

the supply chain?

**•** Retail (0)

**•** Other. Specify: (10)

**•** Very unfamiliar (7)

**•** Somewhat familiar (7)

**•** Unfamiliar (1)

**•** Familiar (8)

**•** Very familiar (14)

**•** Raw/basic material manufacture (1)

**•** Final product manufacture (11)

**•** Warehousing/distribution (14)

**•** Component/sub‐assembly manufacture (1)

**o.** Utilities (1)

**m.** Services: Professional/scientific/technical (12)

**4.** How does your organization conduct supply chain risk analysis? (select one)

**•** We outsource our supply chain risk analysis to a consulting firm. (1)

performs supply chain risk analysis for the organization. (9)

supply chain risk. (Please specify department:) (6)

**•** We have a dedicated Risk Management Department (i.e., an internal consultant) that

**•** Each department is responsible for assessing supply chain risk related to their particular

**•** We have a specific department (e.g., procurement) that is charged with managing

**5.** Which of the following positions best describes your organization's business function in

**6.** How familiar are you with your organization's supply chain risk management process?


**9.** The following statements relate to your organization's quality management system. Please indicate the degree

to which you agree or disagree with each statement in relation to your organization.

(1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, 5 = Strongly agree)

	- **•** Yes (18)
	- **•** No (20)

## **Acknowledgements**

This chapter is built on prior research from "The Relationship of Organizational Quality Management Practice with Supply Chain Risk Management Program Maturity", Thesis for East Carolina University Honors College

## **Author details**

Tyler Florio

Address all correspondence to: tylerwflorio@gmail.com

College of Business, East Carolina University, Greenville, USA

## **References**

**9.** The following statements relate to your organization's quality management system. Please

to which you agree or disagree with each statement in relation to your organization.

(1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, 5 = Strongly

**a.** Customer satisfaction/service data for key indicators are actively accumulated and

**c.** Customer complaints are tracked and managed through a formal complaint system

**e.** Employee use of quality‐related problem solving techniques is actively supported

This chapter is built on prior research from "The Relationship of Organizational Quality Management Practice with Supply Chain Risk Management Program Maturity", Thesis for

**b.** Customer data are used to make internal quality improvements.

indicate the degree

analyzed.

with feedback.

**d.** Customer service standards are kept.

150 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

(ex. DMAIC, control charting, etc.).

**f.** Quality metrics or standards are kept.

**g.** Team‐based quality improvement is actively supported.

**10.** Is your organization certified for ISO 9001 or equivalent?

**h.** Top management has a demonstrated commitment to quality.

agree)

**•** Yes (18)

**•** No (20)

**Author details**

Tyler Florio

**Acknowledgements**

East Carolina University Honors College

Address all correspondence to: tylerwflorio@gmail.com

College of Business, East Carolina University, Greenville, USA


[27] Escanciano C., Fernandez E., Vasquez C., Influence of ISO 9000 certification on the progress of Spanish industry towards TQM. International Journal of Quality & Reliability Management. 2011; 18(5):481–494.

[14] Chacon N., Doherty S., Hayashi C., Green R., New models for addressing supply chain and transport risk: an initiative of the risk response network in collaboration with Accenture. World Economic Forum: Cologny/Geneva Switzerland;

[15] Ghadge A., Dani S., Kalawsky R., Supply chain risk management: present and future scope. International Journal of Logistics Management. 2012; 23(3):313–339.

[16] Kumar S. Managing risks in a relief supply chain in the wake of an adverse event. OR

[17] Aguiar Y.M., Assessing the CARVER+S risk management model of terrorism prepar‐ edness in business continuity planning. Northcentral University, USA: PhD disserta‐

[18] Skipper J.B., Hanna J.B., Gibson B.J., Alabama power response to Katrina: managing a severe service supply chain disruption. Arden: Jordan Whitney Enterprises Inc; 2010.

[19] Ellegaard C., Supply risk management in a small company perspective. Supply Chain

[20] Kern D., Moser R., Hartmann E., Moder M., Supply risk management: model devel‐ opment and empirical analysis. International Journal of Physical Distribution &

[21] Pettit T.J., Supply chain resilience: development of a conceptual framework, an assessment tool and an implementation process. The Ohio State University, USA: PhD

[22] Finch P., Supply chain risk management. Supply Chain Management. 2004; 9(2):183–

[23] Zhao L., Huo B., Sun L., Zhao X., The impact of supply chain risk on supply chain integration and company performance. Supply Chain Management: An International

[24] Fotopoulos C., Psomas E., The use of quality management tools and techniques in ISO 9001:2000 certified companies: the Greek case. International Journal of Productivity and

[25] Psomas E.L., Fotopoulos C.V., Kafetzopoulos D.P., Core process management practices, quality tools and quality improvement in ISO 9001 certified manufacturing companies.

[26] Sampaio P., Saraiva P., Rodrigues A.G., ISO 9001 certification research: questions, answers and approaches. International Journal of Quality & Reliability Management.

2012.

tion; 2011.

dissertation; 2008.

2009; 26(1):38–58.

Journal. 2013; 18(2):115–131.

196.

Insight. 2011; 24(2):131–157.

152 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Management. 2008; 13(6):425–434.

Logistics Management. 2012; 42(1):60–82.

Performance Management. 2009; 58(6):564–580.

Business Process Management Journal. 2011; 17(3):437–460.


#### **ALAMEDA Ecosystem: Centering Efforts in Software Testing Development ALAMEDA Ecosystem: Centering Efforts in Software Testing Development**

José González Enríquez, Julián Alberto García-García, Francisco José Domínguez-Mayo and María José Escalona Cuaresma José González Enríquez, Julián Alberto García-García, Francisco José Domínguez-Mayo and María José Escalona Cuaresma

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66043

#### **Abstract**

[41] Thompson A. Risk Management Framework. Government of South Australia, Ade‐

[42] COSO., Strengthening enterprise risk management for strategic advantage. Committee of Sponsoring Organizations of the Treadway Commission; 2009 [cited 2016 Aug 21].

laide: Department of Communities and Social Inclusion; 2009.

[43] Nunnally J.C., Psychometric theory. 2nd ed. New York: McGraw‐Hill; 1978.

Available from: www.coso.org.

154 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

One of the most important and critical aspects to improve the quality assurance in software is to improve the testing process by utilizing techniques and tools, which will enhance the software testing process, making it more effective and efficient. This chapter presents ALAMEDA ecosystem, a software package that centers its efforts in software testing development and is a result from a real-world project. ALAMEDA provides support to lifecycles focused on the generation, implementation, and testing organization from the earliest stages of software development. In addition, the ecosystem provides an environment of rating the degree of compliance of organizations with the International Standard for Testing ISO/IEC-29119. It is proposed as a tool to use during the various iterations that may occur in an agile software development process.

**Keywords:** testing development, model-driven testing, quality assurance

## **1. Introduction**

The area of Information Technology and Communications (ICT) is an essential element of innovation in other business sectors such as industrial, logistics, health, etc., where the effectiveness and efficiency in software development is critical for a fundamental and safe operation.

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The process of software development involves a series of activities in which the chances of a human error is high (mistakes can happen in the beginning of the process, in which the objectives can be inadequately specified, as well as during later steps). Therefore, it is important to research and define techniques and methods (as well as supporting tools) in order to detect and solve any area of special attention as soon as possible. In this sense, over recent decades, a number of studies have analyzed the cost of resolving software defects. The cost increases with the time of the defect in the system [1]. As such, the cost of rectifying an error is lower, the earlier the problem is detected.

The budget of software projects is critical and limited in many cases [2]. For this reason, both research institutions and enterprises look for solutions to improve quality assurance of the software development process [3]. The finding of methods and tools to improve the quality, reduce costs and increase the guarantee of results becomes an essential aim for enterprises and development teams [4].

One of the most important and critical aspects to improve the quality assurance in software is to improve the testing process by utilizing techniques and tools, which will enhance the software-testing process, making it more robust. It is a systematic method to determine interrelated factors and parameters affecting a process and its desired output [5]. It is usually performed for one of two reasons: to detect defects and to estimate reliability. The key to software testing is trying to find the modes of failure—something that requires testing the code on all possible inputs and depending on the environment it can be implemented at any time during the development process [6].

This chapter presents a testing solution (named ALAMEDA) to innovate in all these aspects related to software testing. ALAMEDA offers to project managers and developers a systematic and tool-based approach for managing (i.e., defining, controlling, measure, etc.) test cases. This systematic and automatic management allows the detection and correction of software errors before they become costly interruptions in the production line.

The ALAMEDA ecosystem aims to provide a set of applications which allows software development companies to ensure the management, automation, and integration of test cases from an early stage of the develop lifecycle. For this purpose, ALAMEDA integrates the wellknown Model-Driven Engineering (MDE) [7, 8] paradigm on software testing and its automation of software and testing development by use of the NDT (Navigational Development Techniques) methodology [9] and NDT-Suite [10].

MDE has been used as a direction vector of the ALAMEDA ecosystem because it is one of most entrenched paradigms within the software engineering area. As such, on the one hand it will help to define the approach and on the other hand, it will manage the conceptual complexity the process engineering area entails. In addition, MDE has been also successfully applied to real software environments and it has shown a considerable area of impact on reducing timeto-market and improving software product quality. For instance, in the business process management [11], healthcare, software product lines [12], or web engineering among others.

The remainder of this chapter is organized as follows: Section 2 presents the theoretical foundations of ALAMEDA. Section 3 presents the ALAMEDA ecosystem for software testing. Finally, Sections 4 and 5 briefly discuss some related work and highlight conclusions.

## **2. Theoretical foundations of ALAMEDA**

The process of software development involves a series of activities in which the chances of a human error is high (mistakes can happen in the beginning of the process, in which the objectives can be inadequately specified, as well as during later steps). Therefore, it is important to research and define techniques and methods (as well as supporting tools) in order to detect and solve any area of special attention as soon as possible. In this sense, over recent decades, a number of studies have analyzed the cost of resolving software defects. The cost increases with the time of the defect in the system [1]. As such, the cost of rectifying an error is lower,

The budget of software projects is critical and limited in many cases [2]. For this reason, both research institutions and enterprises look for solutions to improve quality assurance of the software development process [3]. The finding of methods and tools to improve the quality, reduce costs and increase the guarantee of results becomes an essential aim for enterprises and

One of the most important and critical aspects to improve the quality assurance in software is to improve the testing process by utilizing techniques and tools, which will enhance the software-testing process, making it more robust. It is a systematic method to determine interrelated factors and parameters affecting a process and its desired output [5]. It is usually performed for one of two reasons: to detect defects and to estimate reliability. The key to software testing is trying to find the modes of failure—something that requires testing the code on all possible inputs and depending on the environment it can be implemented at any time

This chapter presents a testing solution (named ALAMEDA) to innovate in all these aspects related to software testing. ALAMEDA offers to project managers and developers a systematic and tool-based approach for managing (i.e., defining, controlling, measure, etc.) test cases. This systematic and automatic management allows the detection and correction of software errors

The ALAMEDA ecosystem aims to provide a set of applications which allows software development companies to ensure the management, automation, and integration of test cases from an early stage of the develop lifecycle. For this purpose, ALAMEDA integrates the wellknown Model-Driven Engineering (MDE) [7, 8] paradigm on software testing and its automation of software and testing development by use of the NDT (Navigational Development

MDE has been used as a direction vector of the ALAMEDA ecosystem because it is one of most entrenched paradigms within the software engineering area. As such, on the one hand it will help to define the approach and on the other hand, it will manage the conceptual complexity the process engineering area entails. In addition, MDE has been also successfully applied to real software environments and it has shown a considerable area of impact on reducing timeto-market and improving software product quality. For instance, in the business process management [11], healthcare, software product lines [12], or web engineering among others.

the earlier the problem is detected.

156 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

during the development process [6].

before they become costly interruptions in the production line.

Techniques) methodology [9] and NDT-Suite [10].

development teams [4].

The model-driven software development has become one of the most important paradigm in computer engineering. Indeed, it has shown a considerable impact on reducing time-to-market and has improved overall quality.

However, the development of high-quality software solutions does not only require systematic processes of software development, but it also requires systematic processes of software testing. At present, this latter aspect has not been attracting attention within the software industry.

It is a fact that if software companies do no invest resources and efforts in the process of software testing, it is quite likely that there will be a waste of money in the short term because of a large volume of errors. Therefore, if a systematic process is not used in tasks of testing, software developers will spend most of the time to check quality and application functionality rather than developing new projects [13].

The ALAMEDA ecosystem aims to innovate in the systematic and automatic application of the process of software testing in the real environment. For this purpose, the ecosystem presented herein takes into account the new standard for software testing (ISO/IEC/IEEE 29119) and a well-validated methodology named NDT. The latter proposes techniques and mechanisms to generate test cases from the early stages of the software lifecycle. These theoretical foundations are explained in the next sections.

#### **2.1. ISO/IEC/IEEE 29119 software testing**

The International Software Testing Standard (ISO/IEC/IEEE 29119) [14] is a standard created between 2013 and 2015. It was created in order to unify all knowledge on software testing and establish basic knowledge concerning this discipline.

On the one hand, this standard provides useful methodological and conceptual guidelines from the perspective of professional or software providers. In addition, it provides (offers) a common terminology, certifications, well-defined techniques, professional qualification, continuous improvement of the process of software testing, inter-operability, and consistency, among others.

On the other hand, and from the perspective of customers, ISO/IEC/IEEE 29119 provides customer confidence on their software provider, an industry benchmark for good practice, and a contractual relationship between the customer and the software provider, among others.

ISO/IEC/IEEE 29119 focuses on a process model of three levels based on the risk factor in the software-testing phase. This model provides guidelines on strategies and policies so as to manage the process of software testing (i.e., defining testing plans, monitoring and controlling software tests, establishing the test environment, executing tests, etc.). All these aspects have been described in detail in five volumes, as follows:


#### **2.2. Model-driven engineering on software testing**

The model-driven engineering paradigm [15] came up in order to tackle the complexity of platforms and the inability of third generation languages to relieve this complexity. It effectively expresses the domain concepts of the problem. This new paradigm, apart from raising the level of abstraction [16], intends to increase automation during the life cycle of software development.

MDE works, as the primary form of expression, with definitions of models (which can be defined, e.g., by means of Unified Modeling Language (UML) [17]) and transformations (which can be defined, e.g., by means of Query/View/Transformation (QVT) [18]) among these models which entail the production of other models. Both elements are essentials to apply MDE. In addition, it is important to take into account the concepts of metamodel and transformation rules. On the one hand, a metamodel is composed of a set of basic elements, its relationships, and its semantic constraints to build well-defined models. On the other hand, a transformation between two models defines a set of relationships between elements of metamodel A (source) and elements of metamodel B (target) [19].

Although MDE offers a theoretical framework, it has not been standardized exactly in these terms in order to facilitate the application of MDE in real projects. OMG presents MDA, which stands for model-driven architecture [20], as a platform to support the MDE paradigm. MDA proposes to base the software development on models which make transformations in order to generate a code or another model with characteristics of a particular technology (or lowest level of abstraction). As transformations go on, it may be noted that the models become more concrete and the abstract model changes into another one compatible with a particular technology or platform. MDA is based on four types of levels or models. These models are (of highest to lowest level of abstraction):

manage the process of software testing (i.e., defining testing plans, monitoring and controlling software tests, establishing the test environment, executing tests, etc.). All these aspects have

**•** ISO/IEC 29119-1: Concepts & Definitions. It introduces the vocabulary on which all standards in the 29119 series are built. It also provides examples of the application of each concept

**•** ISO/IEC 29119-2: Test Processes. It defines a generic process model for software testing that can be used to govern, manage, and implement software testing in any organisation, project, or testing activity. This model is based on three layers: (i) organizational test specifications (e.g., organizational test policy, organizational test strategy); (ii) test management; and (iii)

**•** ISO/IEC 29119-3: Test Documentation. It defines templates for test documentation that cover

**•** ISO/IEC 29119-4: Test Techniques. It defines techniques and methods to design test cases. **•** ISO/IEC 29119-5: Keyword Driven Testing. It defines guidelines for supporting keyworddriven testing which is a way of describing test cases by using a pre-defined set of keywords. These keywords are names which are associated with a set of actions that are required to perform a specific step in a test case. By using keywords to describe test steps instead of natural language, test cases can be made easier to understand, to maintain, and to automate.

The model-driven engineering paradigm [15] came up in order to tackle the complexity of platforms and the inability of third generation languages to relieve this complexity. It effectively expresses the domain concepts of the problem. This new paradigm, apart from raising the level of abstraction [16], intends to increase automation during the life cycle of

MDE works, as the primary form of expression, with definitions of models (which can be defined, e.g., by means of Unified Modeling Language (UML) [17]) and transformations (which can be defined, e.g., by means of Query/View/Transformation (QVT) [18]) among these models which entail the production of other models. Both elements are essentials to apply MDE. In addition, it is important to take into account the concepts of metamodel and transformation rules. On the one hand, a metamodel is composed of a set of basic elements, its relationships, and its semantic constraints to build well-defined models. On the other hand, a transformation between two models defines a set of relationships between elements of metamodel A (source)

Although MDE offers a theoretical framework, it has not been standardized exactly in these terms in order to facilitate the application of MDE in real projects. OMG presents MDA, which stands for model-driven architecture [20], as a platform to support the MDE paradigm. MDA proposes to base the software development on models which make transformations in order to generate a code or another model with characteristics of a particular technology (or lowest

been described in detail in five volumes, as follows:

158 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

in practice.

dynamic testing.

software development.

the entire software testing life cycle.

**2.2. Model-driven engineering on software testing**

and elements of metamodel B (target) [19].


MDE in general (or MDA in particular) has been successfully applied in different environments and research areas [21], software testing is one of them. In fact, this particular case is named as model-driven testing (MDT).

MDT is becoming more standardized in order to approach the automation of software testing, thanks to different research works and proposals [22–24]. This approach can significantly reduce the most painstaking cycle of all software development efforts—testing. Testing currently comprises between 30 and 70% of all software development projects, which calls for a significant amount [25] of resources that must be properly managed. Therefore, MDT and supporting tools enable software developers and testers to become far more productive and reduce the time-to-market, while maintaining high standards of software quality.

**Figure 1.** General scheme for generation of test cases integrated in ALAMEDA.

**Figure 1** shows the vision of MDT from the perspective of MDA. As this suggests, models belonging to the PIM level can be transformed into models belonging to the PSM level, getting the code from the systems derived from PSM through another transformation step. This is the general philosophy of work proposed by MDA, but if this one is applied to software testing, some concepts change. In this case, the process of testing starts with models of System's Test which belong to the PIM level; or also named platform independent test (PIT). The quality team will elaborate on these tests to provide additional information. Next the PSM, which contains specific information about the implementation of the system, can be translated into the Platform Specific Test (PST), using the previous information from PIT and the knowledge of the quality team. Finally, these tests are morphed into codes for testing the system, ensuring the quality of the test and covering all the project-specific requirements.

All these theoretical foundations of MDT have been systematically applied and adapted within the ALAMEDA ecosystem. For this purpose, ALAMEDA implements a methodological framework to generate case tests from functional requirements (FR) as well as managing and controlling these case tests. This framework is based in NDT, a well-contrasted methodology. The foundations of NDT and how this one has been integrated into ALAMEDA are detailed in Section 2.3.

#### **2.3. The NDT methodology from the perspective of software testing**

NDT is a web methodology and is covered by the MDE paradigm. At present, NDT supports classical and agile software development lifecycles. For this purpose, NDT defines a set of metamodels for each phase of the software lifecycle (viability study, requirements, analysis, design, implementation, testing and maintenance) and it establishes a set of transformation rules (based on QVT) to generate models from other ones systematically. This implies a lower cost for the software development because of the automation of the process.

The application of MDE-based methodologies (such as NDT) and particularly, the application of transformations among models may become monotonous and very expensive if there are no supporting tools. These tools automate the process in order to get all the potential of MDE which provides a practical and useful environment to the enterprises. This aspect is one of the virtues of NDT by which this methodology has been applied successfully in many real projects. In this sense, NDT defines a set of supporting tools grouped in NDT-Suite [26]. The main tools in NDT-Suite are: (i) NDT-Profile, which defines UML profiles for each NDT metamodel; (ii) NDT-Quality [27], which measures the quality of use of NDT and checks semantic constraints of NDT; and (iii) NDT-driver [28], which allows running each QVT transformations between models in an automatic manner (for instance, this tool generates test cases from the specification of FRs of a project).

From the perspective of the process of software testing, ALAMEDA uses mechanisms defined by NDT to generate test cases from functional requirements defined in the first phase of the software lifecycle (i.e., the requirement phase). ALAMEDA also proposes how functional testing can be deeply improved by means of *early testing*. This paper aims to briefly present how this mechanism of generation of test cases works [29, 30].

As mentioned above, ALAMEDA implements the concept of *early testing* which means that test cases can be generated from early stages of the software lifecycle. It does not need to define test cases when the system is developed (classic software development lifecycle). In this sense, the generation of test cases of ALAMEDA is based on a four-step process (**Figure 2**):

**Figure 2.** Test generation of ALAMEDA ecosystem.

**Figure 1** shows the vision of MDT from the perspective of MDA. As this suggests, models belonging to the PIM level can be transformed into models belonging to the PSM level, getting the code from the systems derived from PSM through another transformation step. This is the general philosophy of work proposed by MDA, but if this one is applied to software testing, some concepts change. In this case, the process of testing starts with models of System's Test which belong to the PIM level; or also named platform independent test (PIT). The quality team will elaborate on these tests to provide additional information. Next the PSM, which contains specific information about the implementation of the system, can be translated into the Platform Specific Test (PST), using the previous information from PIT and the knowledge of the quality team. Finally, these tests are morphed into codes for testing the system, ensuring

All these theoretical foundations of MDT have been systematically applied and adapted within the ALAMEDA ecosystem. For this purpose, ALAMEDA implements a methodological framework to generate case tests from functional requirements (FR) as well as managing and controlling these case tests. This framework is based in NDT, a well-contrasted methodology. The foundations of NDT and how this one has been integrated into ALAMEDA are detailed

NDT is a web methodology and is covered by the MDE paradigm. At present, NDT supports classical and agile software development lifecycles. For this purpose, NDT defines a set of metamodels for each phase of the software lifecycle (viability study, requirements, analysis, design, implementation, testing and maintenance) and it establishes a set of transformation rules (based on QVT) to generate models from other ones systematically. This implies a lower

The application of MDE-based methodologies (such as NDT) and particularly, the application of transformations among models may become monotonous and very expensive if there are no supporting tools. These tools automate the process in order to get all the potential of MDE which provides a practical and useful environment to the enterprises. This aspect is one of the virtues of NDT by which this methodology has been applied successfully in many real projects. In this sense, NDT defines a set of supporting tools grouped in NDT-Suite [26]. The main tools in NDT-Suite are: (i) NDT-Profile, which defines UML profiles for each NDT metamodel; (ii) NDT-Quality [27], which measures the quality of use of NDT and checks semantic constraints of NDT; and (iii) NDT-driver [28], which allows running each QVT transformations between models in an automatic manner (for instance, this tool generates test cases from the specifica-

From the perspective of the process of software testing, ALAMEDA uses mechanisms defined by NDT to generate test cases from functional requirements defined in the first phase of the software lifecycle (i.e., the requirement phase). ALAMEDA also proposes how functional testing can be deeply improved by means of *early testing*. This paper aims to briefly present

the quality of the test and covering all the project-specific requirements.

160 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**2.3. The NDT methodology from the perspective of software testing**

cost for the software development because of the automation of the process.

how this mechanism of generation of test cases works [29, 30].

in Section 2.3.

tion of FRs of a project).


## **3. ALAMEDA ecosystem**

The ALAMEDA ecosystem for software testing consists of:

**•** a methodological framework based on the ISO 29119 International Testing Standard and

**•** a technological support that allows carrying out of the above methodological framework and is composed of a web platform and a desktop platform. It is worth mentioning at this point that both allow the application of MDT and the automation of development tests by the use of NDT-Suite, respectively. In addition to this, they incorporate a set of tools to automate the dynamic tests and perform the management and quality of their software in an integrated way.

Concretely, the ALAMEDA ecosystem offers a platform as a service (PaaS) ready-to-use, a quick and a progressive methodological implementation and adaptation. It is also flexible and easily scalable and offers an alignment with the ISO 29119 International Testing Standard.

#### **3.1. Methodological framework**

In this section, the different actors and activities involved in the ecosystem are shown. The different cases performed by an actor are also described.

**Figure 3** shows the three main types of actors who inherit different subtypes.

**Figure 3.** Actors of ALAMEDA ecosystem.

	- **◦ AC-02. Director**: Actor who represents a registered user of director or head of the corporation type to ensure product quality software. This actor is divided into two subtypes:
		- **▪ AC-03. Test manager**. Actor who must manage a testing a project and initiating the necessary meetings to ensure its success.
		- **▪ AC-04. Technical manager**. Actor who represents the person with overall responsibility for the management or conduct of the projects concerned.

**◦ AC-05. Tester**. Actor representing the user or users responsible for testing the system. In turn, this actor can be defined as a specific type:

**•** a technological support that allows carrying out of the above methodological framework and is composed of a web platform and a desktop platform. It is worth mentioning at this point that both allow the application of MDT and the automation of development tests by the use of NDT-Suite, respectively. In addition to this, they incorporate a set of tools to automate the dynamic tests and perform the management and quality of their software in

Concretely, the ALAMEDA ecosystem offers a platform as a service (PaaS) ready-to-use, a quick and a progressive methodological implementation and adaptation. It is also flexible and easily scalable and offers an alignment with the ISO 29119 International Testing Standard.

In this section, the different actors and activities involved in the ecosystem are shown. The

**• AC-01. Registered user**: Actor who represents a team member and is also a registered user

**◦ AC-02. Director**: Actor who represents a registered user of director or head of the corporation type to ensure product quality software. This actor is divided into two

**▪ AC-03. Test manager**. Actor who must manage a testing a project and initiating the

**▪ AC-04. Technical manager**. Actor who represents the person with overall responsibil-

ity for the management or conduct of the projects concerned.

of ALAMEDA technological tools. This actor, is nested in:

necessary meetings to ensure its success.

**Figure 3** shows the three main types of actors who inherit different subtypes.

an integrated way.

**3.1. Methodological framework**

**Figure 3.** Actors of ALAMEDA ecosystem.

subtypes:

different cases performed by an actor are also described.

162 Quality Control and Assurance - An Ancient Greek Term Re-Mastered


There are 13 functionalities that define the methodological framework of ALAMEDA ecosystem; those are:


It is important to note that, to perform this step, the "FR-10. Test execution" must be performed by the testers.


important to note that, to carry out this scenario, the "FR-10. Test execution" must be accomplished.


#### **3.2. Software tools ecosystem for testing**

It is important to note that, to perform this step, the "FR-10. Test execution" must be

**• FR-06. Test plan monitoring**. In this scenario, two actors are involved: the test manager and the technical manager. The first thing that the test manager has to do is to manage different tasks for configuring the environment and the test plan and later, to set up the environment for monitoring the test plan. During the process of monitoring, several decisions may be taken. The major duties on the part of the test manager are: (i) report on the progress of the test plan, (ii) control the test plan and report control actions to the technical director. On the side of the technical manager has to: (i) access the ALAMEDA web platform for gathering

**• FR-07. Test plan creation and maintenance**. In this scenario, the actor "tester" in the role of "tester designer" or an "advanced user" is involved. The first action in this scenario is to understand the context of the project. Once understood, the functional requirements of the project must be checked with through EA and NDT-Quality tools. Then, it is necessary to generate the requirements and acceptance tests documented in the TestLink using the ALAMEDA web platform, which is based on the functional requirements defined previously. The next activities that the user must take are to identify and analyze risks and the treatment that will be applied, determine how long the test plan and its programming will

**• FR-08. Design and implement tests**. In this scenario, the actor "tester" in the role of a "tester designer" or an "advanced user" is involved. The first action of this scenario is to generate the test cases using the ALAMEDA web platform. The following actions to be performed: identify feature sets, derive test conditions, derive test coverage items, derive test cases, assemble test sets and finally, derive test procedures. It is important to note that, to perform

**• FR-09. Test environment set-up and maintenance process**. In this scenario, the actor "tester" is involved. For performing this scenario, the test plan and the specification test procedure documents generated in previous steps are necessary. The first action of this scenario is to plan and design the test environment. Then, it has to be properly configured and set up. Finally, it is adamant to prepare data that will be used for the tests. It is important to note that, to perform this step, the "FR-08. Design and implement tests" must be per-

**• FR-10. Test execution**. In this scenario, the actor "tester" is involved. Through the web platform of ALAMEDA ecosystem, the tester will: (i) carry-out the acceptance tests, (ii) check the code's quality of the project, and (iii) perform the functional and stress tests. From the web platform, the user will register the results for comparing them with older versions. It is important to note that, to perform this step, the "FR-09. Test environment set-up and

**• FR-11. Test incident reporting**. In this scenario, the actor "tester" is involved. The first action of this scenario is to analyze the results of the test execution from the web platform. If there are issues, the incident report must be created or updated (where appropriate). It is

this step, the "FR-07. Test plan creation and maintenance " must be performed.

information about control actions and progress of the test plan.

performed by the testers.

164 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

be and finally, record it.

formed.

maintenance process" must be executed.

The software tools that compose the ALAMEDA ecosystem are:


**Table 1** shows how the activities of the methodological framework described before and these tools are implemented in ALAMEDA.



**Table 1.** Description of the implementation of the methodological framework.

#### **3.3. ALAMEDA from a user's perspective**

In this section, functionalities that can be made through the desktop and the web platform of the ALAMEDA ecosystem are described.

#### *3.3.1. Desktop platform*

**Activities Technological support Description of technological support**

report

requirements.

requirements.

Jenkins

JMeter)


Access to the web platform control panel module where user

Access to the web platform to inform about the test execution

Access to the web platform where the indicators related to the

Access to the Enterprise Architect and to the web platform where user will generate tests depending on the defined

Access to Testlink for editing and visualizing test plans.

Access to the Enterprise Architect and to the web platform where user will generate tests depending on the defined

Access to Testlink for editing and visualizing test plans.




Access to the web platform where the indicators related to the


state of the executions or build/s can be displayed.

Access to the web platform where user sill:

Access to the web platform where user sill: - manage the tools configuration. - manage the ISO versions

Access to the web platform where user will: - manage the configuration of ALAMEDA tools

Access to the web platform where user will: - execute load and performance tests Access to Jenkins where user will:

Access to Jenkins where user will: - configure the plugins installed in Jenkins - configure the automated tasks with Jenkins

Access to Testlink where user will:

Access to JMeter where user will: - execute load and performance tests.

will check the status of SonarQuBe executions

state of the executions or build/s can be displayed

Web platform of ALAMEDA ecosystem

166 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Web platform of ALAMEDA ecosystem

Web platform of ALAMEDA ecosystem - Tools to automate dynamic testing and quality Software








**Table 1.** Description of the implementation of the methodological framework.

FR-04. Software quality management

FR-06. Test plan monitoring

FR-07. Test plan creation and maintenance

FR-08. Design and Implement Tests

FR-09. Test Environment Set-Up & Maintenance Process

FR-10. Test execution

FR-11. Test incident reporting

FR-12. Users and services management

FR-13. Tools set-up management

FR-05. Test execution report

> The desktop platform of ALAMEDA ecosystem consists of an assistant for the creation of functional requirements as a preliminary step to the process of generating automated acceptance tests associated with them.

> It is presented as a software layer above deployed EA assisting in the construction of very specific components: actors and system functional requirements. These components are included in EA by previous installation of NDT-Suite and correspond to the basic components to build and validate when creating a specification's requirements model.

The specification's requirements model must follow the structure shown in **Figure 4**.

**Figure 4.** Specification's requirements model.

The construction of the models of functional requirements is based on two operations:


Following the previous structure, steps that are necessary to perform are:

**•** System actor creation/modification. The elements that represent the actors of the system are stereotyped as "AC" and can be found in the EA toolbox provided by NDT-Suite called <NDT System Requirements> DRS Actors. There are two ways of creating these elements:

	- **◦** Extends or include: only applied between RF elements.
	- **◦** Use case: only applied between AC and RF elements.

#### *3.3.2. Web platform*

The web platform of the ALAMEDA ecosystem brings together the bulk of the functionality of ALAMEDA. In this section, the applicability of each module of this platform is addressed.

	- **◦** New user: it allows registering new users.
	- **◦** Edit user: it allows to edit both basic information about users and performs configuration management tools that are assigned to users. Removing users is also allowed.
	- **◦** New self-assessment ISO/IEC-29119: it allows the installation of a new version of the ISO/ IEC-29119 standard. A new version carries a glossary of terms associated with that role and an associated self-assessment questionnaire.
	- **◦** ISO/IEC 29119 management: it allows the management of the versions of the ISO/ IEC-29119 standard loaded in the system, which includes removing and updating previous versions.
	- **◦** Glossary of terms. The glossary of terms shows the collection of terms that defines the norm in the first part for standardizing the terminology used in the field of Testing. In the glossary of terms, each term is shown along with the associated definition.
	- **◦** Self-assessment. The self-assessment consists of a series of questions about compliance with the rule that the user must answer based on what is known in reference to his/her

organization. The format shown in the questions are: "Yes", "No" or "Partly", so that each answer carries a score. The questionnaire is broken down into the parts of the standard and within each part, every field and block is referred to the corresponding question. There is an option for storing the state of the questionnaire and complete it later.

**◦** Drag the item from the toolbox to the corresponding diagram.

**◦** Extends or include: only applied between RF elements.

**◦** Use case: only applied between AC and RF elements.

**◦** New user: it allows registering new users.

and an associated self-assessment questionnaire.

to access to the other sections of the module. These sections are:

option and selecting the actor item.

168 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

*3.3.2. Web platform*

previous versions.

**◦** Select the directory and include the item making right click, choosing the "Add Element"

**•** Functional requirements creation/modification. The elements that represent the functional requirements of the system are stereotyped as "FR" and can be found in EA toolbox provided by NDT-Suite called <NDT System Requirements> DRS Functional Requirements. FRs are the most complex elements of the model; they do not just present a definition of a particular value for a specific property, but rather have collections of properties (pre- and post-conditions) and may be specified based on scenarios (sequence of steps) or activity diagrams. The way the RFs are created are in the same way with the actors one. There are different types of relations between the RF and depend on the side of the connection.

The web platform of the ALAMEDA ecosystem brings together the bulk of the functionality of ALAMEDA. In this section, the applicability of each module of this platform is addressed. **•** Platform management: for access to the management module it is necessary to access as a manager. The administration panel of the platform (**Figure 5**) displays the following options:

**◦** Edit user: it allows to edit both basic information about users and performs configuration

**◦** New self-assessment ISO/IEC-29119: it allows the installation of a new version of the ISO/ IEC-29119 standard. A new version carries a glossary of terms associated with that role

**◦** ISO/IEC 29119 management: it allows the management of the versions of the ISO/ IEC-29119 standard loaded in the system, which includes removing and updating

**•** Compliance with ISO/IEC-29119: the module in accordance with ISO/IEC-29119 standard (**Figure 6**) has four sections; it depends on an existing version of the standard installed on the platform. Therefore, if there is no version installed on the platform, it will not be possible

**◦** Glossary of terms. The glossary of terms shows the collection of terms that defines the norm in the first part for standardizing the terminology used in the field of Testing. In the

**◦** Self-assessment. The self-assessment consists of a series of questions about compliance with the rule that the user must answer based on what is known in reference to his/her

glossary of terms, each term is shown along with the associated definition.

management tools that are assigned to users. Removing users is also allowed.

	- **◦** Tests generation. The test generation process receives as input a specification requirements model in EA format and it returns a test plan generated in the TestLink tool with a build that allows its execution in this tool. The build process begins with a validation phase input model. If errors are found in the model definition, the test generation is not addressed. Instead, a list with the errors found during the process is provided.
	- **◦** Generated tests display. The result of the test generation process, regardless of whether they have been generated with errors or not during the validation phase, are displayed in the result viewer.
	- **◦** Tasks generation. The process consists of a scanning of the test cases executed in TestLink to generate tasks associated to the Redmine management tool.
	- **◦** Tasks monitoring and visualization. The result viewer is similar to the one shown in the test generation module with the difference that the tool deployed is Redmine. The available project tree corresponds to the generated tests. From the generated tests, the viewer can carry out the task monitoring.
	- **◦** Static code analysis. The tool used to perform static code analysis using the web platform is SonaQube. The access form to the execution of static code analysis requests two inputs to perform the test: the project to analyze and its version. The use of compressed projects files as rar or zip format is allowed.



**Figure 5.** Management platform.

**Figure 6.** Compliance with ISO/IEC-29119.

**◦** measure the quality of the code source of software products through static tests (Sonar-

**◦** execute acceptance tests previously generated in the test execution module (Testlink).

**•** Furthermore, as in the test generation module, a result viewer to monitor and track all the

**◦** Static code analysis. The tool used to perform static code analysis using the web platform is SonaQube. The access form to the execution of static code analysis requests two inputs to perform the test: the project to analyze and its version. The use of compressed projects

**•** Control panel. This module allows the monitoring of the elements generated in all the

**◦** perform dynamic tests in order to monitor the web nature software product measurement

Qube tool).

(JMeter).

conducted activities is provided.

files as rar or zip format is allowed.

170 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

different modules of the web platform.

**Figure 5.** Management platform.


**Figure 7.** Test generation.


**Figure 8.** Software and testing analysis.

## **4. Conclusions and future work**

This chapter presents the ALAMEDA ecosystem for software testing. Over the last decade, a number of studies, as well as technical and theoretical proposals have emerged to improve the process of software testing. However, the software industry has often carried out this process in a manner that was not standardized and hence, proved difficult to imply to systematization, management and continuous improvement of this process.

The ALAMEDA platform has been developed so as to cover the observed gap in the industry. In this sense, ALAMEDA provides a methodological framework based on the ISO/IEC 29119 standard and the application of the MDE paradigm of software testing (known as MDT).

Regarding the ISO/IEC 29119, this one is a new standard which implies a certain novelty of the software industry. ALAMEDA integrates the main foundations of this standard within its ecosystem in order to facilitate the adoption of this standard by industry. In addition, ALA-MEDA provides mechanisms to measure the degree of compliance to the ISO/IEC 29119.Thus, any software company can measure its evolution to meet this standard and carry out a process of continuous improvement in its software development processes.

Regarding the implementation of MDT, ALAMEDA has included the concept of early testing, thanks to it for integrating the NDT methodology within its ecosystem. Thus, it is possible to obtain test cases from the functional requirements and manage these tests effectively. Moreover, this mechanism provides traceability between tests and functional requirements allowing always to know which functionality is being tested.

Finally, and taking into account the features of ALAMEDA, it has not been possible to find a technological solution to cater for all needs associated with the process of software testing from the early stages and complying with the international standard ISO/IEC 29119. In this sense, ALAMEDA improves the state-of-the-art within the management of software quality assurance because it provides an innovative technological solution which is based on well-validated theoretical foundations.

## **Acknowledgements**

This research has been supported by the MeGUS project (TIN2013-46928-C3-3-R), by the SoftPLM Network (TIN2015-71938-REDT) of the Spanish Ministry of Economy and Competitiveness and Fujitsu Laboratories of Europe (FLE). In addition, we would like to acknowledge the leadership of Servinform during the development of the ALAMEDA ecosystem. Our research group has collaborated in the conceptual design of ALAMEDA as well as in analysis, design, and testing tasks of the platform.

## **Author details**

**Figure 8.** Software and testing analysis.

**4. Conclusions and future work**

172 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

management and continuous improvement of this process.

of continuous improvement in its software development processes.

always to know which functionality is being tested.

theoretical foundations.

This chapter presents the ALAMEDA ecosystem for software testing. Over the last decade, a number of studies, as well as technical and theoretical proposals have emerged to improve the process of software testing. However, the software industry has often carried out this process in a manner that was not standardized and hence, proved difficult to imply to systematization,

The ALAMEDA platform has been developed so as to cover the observed gap in the industry. In this sense, ALAMEDA provides a methodological framework based on the ISO/IEC 29119 standard and the application of the MDE paradigm of software testing (known as MDT).

Regarding the ISO/IEC 29119, this one is a new standard which implies a certain novelty of the software industry. ALAMEDA integrates the main foundations of this standard within its ecosystem in order to facilitate the adoption of this standard by industry. In addition, ALA-MEDA provides mechanisms to measure the degree of compliance to the ISO/IEC 29119.Thus, any software company can measure its evolution to meet this standard and carry out a process

Regarding the implementation of MDT, ALAMEDA has included the concept of early testing, thanks to it for integrating the NDT methodology within its ecosystem. Thus, it is possible to obtain test cases from the functional requirements and manage these tests effectively. Moreover, this mechanism provides traceability between tests and functional requirements allowing

Finally, and taking into account the features of ALAMEDA, it has not been possible to find a technological solution to cater for all needs associated with the process of software testing from the early stages and complying with the international standard ISO/IEC 29119. In this sense, ALAMEDA improves the state-of-the-art within the management of software quality assurance because it provides an innovative technological solution which is based on well-validated José González Enríquez\* , Julián Alberto García-García, Francisco José Domínguez-Mayo and María José Escalona Cuaresma

\*Address all correspondence to: jose.gonzalez@iwt2.org

Computer Languages and Systems Department, University of Seville, Seville, Spain

## **References**


[24] Dai ZR. (2004). Model-Driven Testing with UML 2.0. In Proceedings of the 2nd European Workshop on Model Driven Architecture, Computing Laboratory, University of Kent, Canterbury, UK, 2004, pp. 179–187

[9] Miller J & Mukerji J. (2003). Model Driven Architecture 1.0.1 Guide. Object Management

[10] Escalona MJ & Aragón G. (2008). NDT. A Model-Driven Approach for Web Require-

[11] García-García JA, Ortega MA, García-Borgoñon L & Escalona MJ. (2012). NDT-Suite: A Model-Based Suite for the Application of NDT. In Web Engineering. Springer, Berlin

[12] García-García JA, Escalona MJ, Martinez-Garcia A, Parra C & Wojdyeski T. (2015). Clinical Process Management: A Model-Driven & Tool-Based Proposal. In Information Systems Development: Transforming Healthcare through Information Systems.

[13] Tassey G. (2002). The Economic Impacts of Inadequate Infrastructure for Software Testing. National Institute of Standards and Technology, RTI Project, 7007(011).

[14] López AS, Valle DC, Escalona MJ, Lee V & Goto M. (2015). Patient Lifecycle Management: An Approach for Clinical Processes. In Bioinformatics and Biomedical Engineer-

[15] Kent S. (2002, May). Model Driven Engineering. In International Conference on

[16] ISO: ISO/IEC/IEEE 29119 Software Testing. http://www.softwaretestingstandard.org/

[17] Fondemenet F & Silaghi R. (2004). Defining Model Driven Engineering Process. 3rd Workshop in Software Model Engineering (WISME2004). October 11–15, Lisbon,

[18] OMG. (2005). UML2.0, Unified Modelling Language. Object Management Group,

[19] OMG. (2008). Documents Associated with Meta Object Facility (MOF) 2.0 Query/View/ Transformation. Object Management Group. http://www.omg.org/spec/QVT/1.0/.

[20] Thiry L & Thirion B. (2009). Functional Metamodels for Systems and Software. Journal of Systems and Software. 82(7):1125–1136. DOI: http://dx.doi.org/10.1016/j.jss.

[21] Whittle J, Hutchinson J & Rouncefield M. (2014). The State of Practice in Model-Driven

[22] OMG. (2003). MDA Guide of OMG. Version 1.0.1, http://www.omg.org/docs/omg/

[23] Object Management Group (OMG). http://utp.omg.org/. Last accessed: July, 2016

Integrated Formal Methods. Springer, Berlin Heidelberg. pp. 286–298.

ments. IEEE Transactions on Software Engineering. 34:377–394.

ing. Springer International Publishing. pp. 694–700.

Group, Inc.

Heidelberg. pp. 469–472.

174 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

(2013). Accessed 10 May 2016.

Portugal. 2004.

2009.01.042. 2009.

03-06-01.pdf

Engineering. IEEE Software. 31(3):79–85.

2008.

formal/2005-07-05, 2005.


## **Improving Quality Assurance in Multidisciplinary Engineering Environments with Semantic Technologies Improving Quality Assurance in Multidisciplinary Engineering Environments with Semantic Technologies**

Dietmar Winkler, Marta Sabou and Stefan Biffl Dietmar Winkler, Marta Sabou and Stefan Biffl

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66222

#### **Abstract**

In multidisciplinary engineering (MDE) projects, for example, automation systems or manufacturing systems, stakeholders from various disciplines, for example, electrics, mechanics and software, have to collaborate. In industry practice, engineers apply individual and highly specialized tools with strong limitation regarding defect detection in early engineering phases. Experts typically execute reviews with limited tool support which make engineering projects defective and risky. Semantic Web Technologies (SWTs) can help to bridge the gap between heterogeneous sources as foundation for efficient and effective defect detection. Main questions focus on (a) how to bridge gaps between loosely coupled tools and incompatible data models and (b) how SWTs can help to support efficient and effective defect detection in context of engineering process improvement. This chapter describes success-critical requirements for defect detection in MDE and shows how SWTs can provide the foundation for early and efficient defect detection with an adapted review approach. The proposed defect detection framework (DDF) suggests different levels of SWT contributions as a roadmap for engineering process improvement. Two selected industry-related real-life cases show different levels of SWT involvement. Although SWTs have been successfully applied in real-life use cases, SWT applications can be risky if applied without good understanding of success factors and limitations.

**Keywords:** Semantic Web Technologies, automation systems engineering, multidisciplinary engineering, quality assurance, defect detection, engineering process improvement

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## **1. Introduction**

In multidisciplinary engineering (MDE) environments, engineers coming from different disciplines have to collaborate and exchange data. For instance, in automation systems engineering (ASE) projects, mechanical, electrical, and software engineering disciplines work together to build manufacturing plants, power plants, or steel mills [1]. However, engineers typically use isolated tools with data models that provide only limited capabilities regarding data exchange and synchronization of engineering plans across disciplines and organizations [2]. Such loosely coupled tools and incompatible data models hinder efficient engineering processes, project management, quality assurance, and engineering process improvement [3]. Due to limited collaboration capabilities, engineering projects become risky and error-prone.

However, the collaboration of different engineering disciplines requires efficient mechanisms for quality assurance and defect detection [4]. In the context of this paper, defects represent errors in engineering plans and design documents, deviations and inconsistencies between engineering plans. Established tool suites, that is, all-in-one solutions, such as EPlan Engineering Configuration (EEC)1 or COMOS2 , with integrated data models typically include basic quality assurance mechanisms, such as syntax checks or basic consistency checks to identify intra-disciplinary defects (i.e., defects within one engineering artifact or between artifacts within one discipline). It is worth noting that there is no strong support for identifying interdisciplinary defects, that is, defects that affect two or more disciplines. Thus, in tool networks, where no integrated data models are available, further research is required to enable effective and efficient defect detection. Performed research revealed that there is very limited support for defect detection in heterogeneous environments. The latter include but are not limited to identifying inconsistencies between variables of mechanical and software (control) engineering artifacts because of the technical heterogeneity of tools and the semantic heterogeneity of data models [5].

The application of reviews [6] (or software inspection) in software engineering supports early defect detection [6], typically executed by human experts in a paper-based way, that is, without or with limited tool support. Thus, defect detection requires a high effort and includes risks to oversee important defects, even in homogenous engineering artifacts such as design documents or software code. In distributed and heterogeneous engineering environments, reviews require additional knowledge of human experts, who are sufficiently familiar with at least two related disciplines and require high cognitive skills. Again, review tasks are often executed manually by these experts and require as such considerable effort. It is worth mentioning that preliminary research indicates that they often miss important defects [7], especially if more than one engineering discipline is involved. Thus, an important issue is the heterogeneity of data models, embodied within applied tools.

Semantic Web Technologies (SWTs) can provide concepts for closing the gap between data models for data management, that is, the identification of data entities and knowledge, data

<sup>1</sup> EPlan: www.eplan.de/.

<sup>2</sup> COMOS: w3.siemens.com/mcms/plant-engineering-software/de/Seiten/Default.aspx.

organization, storage, and querying for project analysis purposes [8]. Thus, the application of SWTs can provide the foundation for effective and efficient defect detection mechanisms. However, even without using SWTs, systematic defect detection approaches, such as (software) reviews or inspection [9], can support defect detection processes, by applying guidelines and reading techniques for defect detection support. Thus, the main research challenges focus on (a) providing expert support for identifying defects in an interdisciplinary context more effectively and efficiently and (b) improving defect detection in organizations with or without SWTs. A defect detection framework (DDF) aims at providing an approach to provide the degree of SWT contributions from solution approaches without SWT (lowest level) to a fully supported SWT solution (highest level). For every level, this chapter describes a prototype solution of a real-life use case, developed together with industry and research experts for evaluation purposes.

Thus, the main goals of this chapter focus on (a) providing a defect detection framework (DDF) to support the introduction of SWTs in an organization on various levels of granularity and (b) to provide lessons learned from industry prototypes in context of SWT applications for defect detection in industrial and research real-life use cases. Further, the chapter demonstrates different levels of SWT capabilities in industry use cases and report on lessons learned on the strengths and limitations of these approaches.

The remainder of this paper is structured as follows: Section 2 presents related work on multidisciplinary engineering (MDE), defect detection, and Semantic Web Technologies. Section 3 motivates research issues, and Section 4 presents the solution approach, that is, requirements and capabilities for defect detection in context of MDE and SWT and the defect detection framework with and without SWTs. Section 5 discusses industry (real-life) use cases and prototype implementations regarding defect detection capabilities, and Section 6 discusses qualitative assessment results and lessons learned on the limitations of individual levels of semantic integration.

## **2. Related work**

**1. Introduction**

178 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

neering Configuration (EEC)1

geneity of data models [5].

1

2

EPlan: www.eplan.de/.

data models, embodied within applied tools.

COMOS: w3.siemens.com/mcms/plant-engineering-software/de/Seiten/Default.aspx.

In multidisciplinary engineering (MDE) environments, engineers coming from different disciplines have to collaborate and exchange data. For instance, in automation systems engineering (ASE) projects, mechanical, electrical, and software engineering disciplines work together to build manufacturing plants, power plants, or steel mills [1]. However, engineers typically use isolated tools with data models that provide only limited capabilities regarding data exchange and synchronization of engineering plans across disciplines and organizations [2]. Such loosely coupled tools and incompatible data models hinder efficient engineering processes, project management, quality assurance, and engineering process improvement [3]. Due to limited collaboration capabilities, engineering projects become risky and error-prone. However, the collaboration of different engineering disciplines requires efficient mechanisms for quality assurance and defect detection [4]. In the context of this paper, defects represent errors in engineering plans and design documents, deviations and inconsistencies between engineering plans. Established tool suites, that is, all-in-one solutions, such as EPlan Engi-

quality assurance mechanisms, such as syntax checks or basic consistency checks to identify intra-disciplinary defects (i.e., defects within one engineering artifact or between artifacts within one discipline). It is worth noting that there is no strong support for identifying interdisciplinary defects, that is, defects that affect two or more disciplines. Thus, in tool networks, where no integrated data models are available, further research is required to enable effective and efficient defect detection. Performed research revealed that there is very limited support for defect detection in heterogeneous environments. The latter include but are not limited to identifying inconsistencies between variables of mechanical and software (control) engineering artifacts because of the technical heterogeneity of tools and the semantic hetero-

The application of reviews [6] (or software inspection) in software engineering supports early defect detection [6], typically executed by human experts in a paper-based way, that is, without or with limited tool support. Thus, defect detection requires a high effort and includes risks to oversee important defects, even in homogenous engineering artifacts such as design documents or software code. In distributed and heterogeneous engineering environments, reviews require additional knowledge of human experts, who are sufficiently familiar with at least two related disciplines and require high cognitive skills. Again, review tasks are often executed manually by these experts and require as such considerable effort. It is worth mentioning that preliminary research indicates that they often miss important defects [7], especially if more than one engineering discipline is involved. Thus, an important issue is the heterogeneity of

Semantic Web Technologies (SWTs) can provide concepts for closing the gap between data models for data management, that is, the identification of data entities and knowledge, data

, with integrated data models typically include basic

or COMOS2

This section summarizes related work on multidisciplinary engineering (MDE) environments, defect detection, and Semantic Web Technologies as foundation for defect detection tool prototypes in context of the defect detection framework (DDF) with and without SWTs.

#### **2.1. Multidisciplinary engineering environments and automation systems engineering**

In *multidisciplinary engineering (MDE)* environments for *automation systems engineering (ASE)*, various stakeholders coming from different disciplines have to collaborate along the automation systems life cycle [1, 10]. Typical examples of such MDE environments for ASE include hydro power plant engineering, steel mill engineering, or industrial production systems. Common to all application areas is the contribution of heterogeneous engineering disciplines, that is, mechanical, electrical, and software engineering, with specialized engineering tools and related specific and heterogeneous data models. However, these expert tools typically have limitations regarding seamless data exchange and collaboration support of engineers [1, 2, 11], a challenge for effective and efficient defect detection. Defects are typically considered as deviations between engineering plans of heterogeneous disciplines, for example, a mismatch of variable definitions and their usage, missing components, or wrong components. Yet, these deviations could be real defects or required and intended changes, requested by an engineer of one specific discipline. For example, the exchange of a hardware sensor from analogue to a digital (required by the electrical engineer) might result in different numbers of pins that need to be addressed by the software engineer in the control software. This change needs to be identified and analyzed early in the engineering process. Changes that arise late in the engineering project, during the commissioning phase, typically require significantly higher efforts for rework [12], during the commissioning phase.

**Figure 1** presents a typical sequential engineering process approach in the ASE domain with parallel engineering activities along the project course. Furthermore, selected and important engineering artifacts of leading engineering disciplines and isolated and distributed quality assurance activities for individual process steps are included in this sample process. In common industry projects, engineers often follow such a simplified sequential engineering process [1, 4], that is, system design, implementation, test and commissioning, and operation (**Figure 1**). Changes and defects can have a critical impact on products, engineering projects, and processes, as they are likely to propagate from early to later stages. Therefore, a need arises for early support of defect detection mechanisms in MDE for ASE projects.

**Figure 1.** Typical sequential engineering process approach with parallel engineering activities [13].

Synchronization and data exchange in an engineering tool chain are typically executed manually or semi-automatically by applying local supporting tools (e.g., based on local database solutions or spread sheet solutions) [13]. However, these approaches incur high maintenance effort and assume the availability of experts, who are responsible for operating and maintaining these tools (often as add-on activities to their primary work tasks). Thus, there is a need for mechanisms that provide capabilities for mapping heterogeneous data models and for supporting defect detection processes across tools, domains, and data models. To overcome semantically heterogeneous data models, a common data exchange format could bridge the semantic gaps between engineering disciplines. The data format needs to be defined by consideration of all related tools; data to be exchanged need to be mapped to pass information and changes from one discipline (and data model) to the other. To overcome individual effort for data format definition and mapping, a standardized data exchange format is reasonable. *AutomationML* [14–16] is such an emerging data exchange standard that supports efficient synchronization of data produced by different engineering disciplines. However, *AutomationML* represents a data exchange standard and needs additional components that support data exchange, synchronization, and defect detection. The *AML.hub*<sup>3</sup> [17] is a platform for providing efficient data exchange/synchronization based on *AutomationML*. Although the *AML.hub* provides mechanisms for data exchange (e.g., mapping and merging), there is currently no support for quality assurance and defect detection.

#### **2.2. Defect detection in software and systems engineering**

tools typically have limitations regarding seamless data exchange and collaboration support of engineers [1, 2, 11], a challenge for effective and efficient defect detection. Defects are typically considered as deviations between engineering plans of heterogeneous disciplines, for example, a mismatch of variable definitions and their usage, missing components, or wrong components. Yet, these deviations could be real defects or required and intended changes, requested by an engineer of one specific discipline. For example, the exchange of a hardware sensor from analogue to a digital (required by the electrical engineer) might result in different numbers of pins that need to be addressed by the software engineer in the control software. This change needs to be identified and analyzed early in the engineering process. Changes that arise late in the engineering project, during the commissioning phase, typically require significantly higher efforts for rework [12], during the commissioning phase.

180 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**Figure 1** presents a typical sequential engineering process approach in the ASE domain with parallel engineering activities along the project course. Furthermore, selected and important engineering artifacts of leading engineering disciplines and isolated and distributed quality assurance activities for individual process steps are included in this sample process. In common industry projects, engineers often follow such a simplified sequential engineering process [1, 4], that is, system design, implementation, test and commissioning, and operation (**Figure 1**). Changes and defects can have a critical impact on products, engineering projects, and processes, as they are likely to propagate from early to later stages. Therefore, a need arises

for early support of defect detection mechanisms in MDE for ASE projects.

**Figure 1.** Typical sequential engineering process approach with parallel engineering activities [13].

Early *defect detection* is a key capability in engineering projects. In software engineering, software reviews are well established to identify defects in software artifacts early and efficiently [9, 18]. These reviews and inspections follow a defect detection process and enable the systematic identification of defects by single reviewers or a review team. Reading techniques are established approaches to support defect detection processes by providing guidelines on how to traverse and read an artifact under review [19]. For example, *scenario-based reading* supports the review of artifacts based on success-critical scenarios or business cases. The main advantage is the focus on real-life use cases that can be prioritized according to engineering risks or business value. *Perspective-based reading* takes the perspectives of different engineering roles, for example, systems architecture, test, or user perspectives to identify defects from various viewpoints. The primary advantage of this approach is the ability of the approach to identify defects from different perspectives based on individual experiences of the experts. In MDE environments, where engineers come from different disciplines, these disciplines may offer useful perspectives for reviewing engineering artifacts to find a variety of different defects.

However, in MDE environments review approaches are often conducted manually by experts. Thus, traditional reviews are effort consuming, expensive, and error-prone because reviewers can easily oversee related defects in engineering plans [20]. Mechanisms with tool-supported defect detection [21] aim to guide reviewers through the review process by focusing on most

<sup>3</sup> AML.hub video: https://www.youtube.com/watch?v=nPg66g46-eM&feature=youtu.be, access: 2016-09.

relevant aspects or critical system characteristics of the engineering artifacts [22]. In software engineering, *Gerrit Code Review*<sup>4</sup> is a widely used review approach for code reviews. Its main features include the comparison of two code variants (e.g., newly available/changed code components and previously developed code component versions, typically stored in a central code repository, GIT5 ), commenting of code fragments, and decision support (e.g., accepting or rejecting modifications). However, *Gerrit Code Review* does not provide any specific guidelines or process support for defect detection in code documents and is focused on software code or applicable on (structured) text elements. In context of defect detection in MDE environments, this approach is comparable to the *Focused Inspection approach* [23] that takes as input deviations or change analysis results between engineering plans to identify changes/ defects from the perspective of mechanical, electrical, or software engineers. In order to reduce the manual effort in reviewing activities, SWT-based mechanisms can provide tool support for defect detection. Other tool solutions, such as *DefectRadar*<sup>6</sup> , support defect detection by providing annotations and extensive commenting. *DefectRadar* has been applied in building automation without connection to software and/or systems engineering. As such, within the context of review support, the concept of *DefectRadar* can help to better identify entities, relationships, and attributes in different types of documents (e.g., engineering plans or models) to enhance understanding of the artifacts and to support defect detection. To the best of the authors' knowledge, there exists no tool that strongly supports defect detection in MDE environments in a comprehensive way. It is worth noting that Semantic Web Technologies (SWT) represent a promising approach to address these gaps and to support defect detection in MDE environments.

#### **2.3. Semantic Web Technologies for defect detection**

*Semantic Web Technologies* (SWTs) were used in several works to support consistency management across engineering models, including defect detection. Feldmann *et al*. [5, 25] focus on identifying inconsistencies that may arise among diverse engineering plans and engineering models created during the ASE process. Such inconsistency detection contributes to the increased productivity of the engineering process as it supports the detection of potentially severe defects early in the engineering process. The *resource description framework* (RDF) is used to uniformly represent engineering models, which are then queried with SPARQL,7 a query language for ontologies similar to SQL for databases, to detect defects. Kovalenko *et al*. [26] present an ontology-based approach to automatically detect inconsistencies across heterogeneous engineering data sets. Ontologies are used to explicitly represent the discipline-/toolspecific knowledge and data in a machine-understandable form. Mappings are then defined between the ontologies to model cross-disciplinary (or cross-tool) relations between the data models and data sets explicit for knowledge integration. SPARQL queries are executed over the discipline/tool ontologies regarding the defined mappings in order to perform inconsis-

<sup>4</sup> Google Gerrit: https://www.Gerritcodereview.com, access 2016-09.

<sup>5</sup> GIT: https://git-scm.com/, access 2016-09.

<sup>6</sup> DefectRadar: https://www.defectradar.com/de, accessed 2016-09.

<sup>7</sup> SPARQL: www.w3.org/TR/rdf-sparql-query/, access 2016-09.

tency detection across discipline/tool boundaries. An approach for the automated validation of plant models is presented in [27], where CAEX models are transformed to ontology-based representations and verified through SPARQL queries. In the area of requirements and test case management, Feldmann et al. [28] present a modeling approach that enables the early integration of requirements and test cases. Furthermore, the authors present a case study based on Semantic Web Technologies (SWTs) to ensure the consistency of requirements as well as of requirements and test cases. A conceptual model that describes the main elements for this use case is developed and then formalized as an ontology. Reasoning mechanisms are applied to support various consistency checks-related cases and requirements. Basically, these works show how SWTs can be useful for defect detection in the automation systems domain.

## **3. Research issues and approach**

relevant aspects or critical system characteristics of the engineering artifacts [22]. In software

features include the comparison of two code variants (e.g., newly available/changed code components and previously developed code component versions, typically stored in a central

or rejecting modifications). However, *Gerrit Code Review* does not provide any specific guidelines or process support for defect detection in code documents and is focused on software code or applicable on (structured) text elements. In context of defect detection in MDE environments, this approach is comparable to the *Focused Inspection approach* [23] that takes as input deviations or change analysis results between engineering plans to identify changes/ defects from the perspective of mechanical, electrical, or software engineers. In order to reduce the manual effort in reviewing activities, SWT-based mechanisms can provide tool support for

providing annotations and extensive commenting. *DefectRadar* has been applied in building automation without connection to software and/or systems engineering. As such, within the context of review support, the concept of *DefectRadar* can help to better identify entities, relationships, and attributes in different types of documents (e.g., engineering plans or models) to enhance understanding of the artifacts and to support defect detection. To the best of the authors' knowledge, there exists no tool that strongly supports defect detection in MDE environments in a comprehensive way. It is worth noting that Semantic Web Technologies (SWT) represent a promising approach to address these gaps and to support defect detection

*Semantic Web Technologies* (SWTs) were used in several works to support consistency management across engineering models, including defect detection. Feldmann *et al*. [5, 25] focus on identifying inconsistencies that may arise among diverse engineering plans and engineering models created during the ASE process. Such inconsistency detection contributes to the increased productivity of the engineering process as it supports the detection of potentially severe defects early in the engineering process. The *resource description framework* (RDF) is used

language for ontologies similar to SQL for databases, to detect defects. Kovalenko *et al*. [26] present an ontology-based approach to automatically detect inconsistencies across heterogeneous engineering data sets. Ontologies are used to explicitly represent the discipline-/toolspecific knowledge and data in a machine-understandable form. Mappings are then defined between the ontologies to model cross-disciplinary (or cross-tool) relations between the data models and data sets explicit for knowledge integration. SPARQL queries are executed over the discipline/tool ontologies regarding the defined mappings in order to perform inconsis-

to uniformly represent engineering models, which are then queried with SPARQL,7

defect detection. Other tool solutions, such as *DefectRadar*<sup>6</sup>

**2.3. Semantic Web Technologies for defect detection**

Google Gerrit: https://www.Gerritcodereview.com, access 2016-09.

DefectRadar: https://www.defectradar.com/de, accessed 2016-09.

SPARQL: www.w3.org/TR/rdf-sparql-query/, access 2016-09.

GIT: https://git-scm.com/, access 2016-09.

is a widely used review approach for code reviews. Its main

, support defect detection by

a query

), commenting of code fragments, and decision support (e.g., accepting

engineering, *Gerrit Code Review*<sup>4</sup>

182 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

code repository, GIT5

in MDE environments.

4

5

6

7

In the context of *multidisciplinary engineering (MDE)* for *automation systems engineering* (ASE), strong limitations regarding early and efficient *defect detection* for distributed tools and heterogeneous data models were observed, where SWTs can help to semantically integrate these heterogeneous data models. An important question is, therefore, how well SWT mechanisms can help improve defect detection. From this fundamental question, the following research issues are derived:

*RI.1: What are success-critical requirements in MDE to enable early, effective, and efficient defect detection to reduce project risks?* Based on observations and literature, this chapter will identify a set of success-critical requirements to support managers and engineers in identifying defects early, effective, and efficient.

*RI.2: How can SWT contributions be organized in a defect detection framework (DDF) to support defect detection on different levels?* Different levels of SWT contributions include defect detection without SWT, defect detection with common concepts (basic approach), and defect detection based on semantically integrated data (advanced approach). However, different levels of SWT contribution can require process changes, additional effort for implementation and application, knowledge experts for SWT implementation, and may need new/adapted methodological approaches and tooling support.

The main question is how this framework can be used to introduce SWT-driven defect detection in organizations and how to establish an appropriate improvement strategy.

*RI.3: What are the benefits and limitations of selected industrial cases and prototype implementations with respect to SWT contributions and defect detection in MDE environments?* In the context of the CDL-Flex8 research laboratory, researchers and industry collaborators developed application scenarios and real-life cases that focus on engineering process improvement and defect detection in industry contexts. Selected industry prototypes are presented for every level of the DDF including discussions on the benefits and limitations in context of requirements and

<sup>8</sup> CDL-Flex: Christian Doppler Laboratory for Software Engineering Integration for Flexible Automation Systems, http:// cdl.ifs.tuwien.ac.at.

expected capabilities. Thus, RI.3 discusses results of prototype solutions and the strengths and limitations of SWT mechanisms in the context of an MDE development processes.

## **4. Defect detection framework with/without SWTs**

This section summarizes basic requirements, needed defect detection capabilities, and introduces the defect detection framework (DDF) for assessing the level of SWT contributions for defect detection in MDE environments.

#### **4.1. Requirements for defect detection in MDE environments**

Requirements and key capabilities have been derived by industry and research experts as a foundation for supporting (a) the development of the defect detection framework and (b) for providing key requirements for offering tool support for defect detection. Basically, requirements include four different core topics: process capabilities (ability to support systematic, traceable, and repeatable processes from quality assurance perspective), organizational requirements (for organizing activities and engineering knowledge), defect detection capabilities (i.e., the core component for defect detection), and tool capabilities that focus on needs for defect detection tool support.

**Process capabilities** include process support for defect detection, embedded within engineering processes:


**Organizational requirements** focus on company and organization issues as prerequisites for implementing (and applying) defect detection approaches.


**Defect detection performance** represents the core capability for defect detection with focus on defect detection performance (i.e., efficiency and effectiveness) and the effort for applying the method/tool.


**Tool capabilities** include basic requirements for a tool solution (on different levels) to support domain experts in the context of defect detection.


The implementation of SWT mechanisms can make data exchange and synchronization of heterogeneous data models more effective and efficient and can enable effective and efficient defect detection.

#### **4.2. Defect detection framework concept**

expected capabilities. Thus, RI.3 discusses results of prototype solutions and the strengths and

This section summarizes basic requirements, needed defect detection capabilities, and introduces the defect detection framework (DDF) for assessing the level of SWT contributions

Requirements and key capabilities have been derived by industry and research experts as a foundation for supporting (a) the development of the defect detection framework and (b) for providing key requirements for offering tool support for defect detection. Basically, requirements include four different core topics: process capabilities (ability to support systematic, traceable, and repeatable processes from quality assurance perspective), organizational requirements (for organizing activities and engineering knowledge), defect detection capabilities (i.e., the core component for defect detection), and tool capabilities that focus on needs

**Process capabilities** include process support for defect detection, embedded within engineer-

**•** *Systematic defect detection processes* to enable repeatability and traceability of defects and

**•** *Traceability* of quality assurance activities focuses on traceability of quality assurance processes, for example, review processes, and results, that is, defects, in various engineering

**Organizational requirements** focus on company and organization issues as prerequisites for

**•** *Effort for method implementation* is an important factor in context of method implementation

**•** *Knowledge and skills* needed. Within the context of SWT, knowledge experts can be required to support method implementation and maintenance. These expert efforts typically result

**Defect detection performance** represents the core capability for defect detection with focus on defect detection performance (i.e., efficiency and effectiveness) and the effort for applying

limitations of SWT mechanisms in the context of an MDE development processes.

**4. Defect detection framework with/without SWTs**

**4.1. Requirements for defect detection in MDE environments**

implementing (and applying) defect detection approaches.

because of trade-off considerations of benefits and costs.

in additional costs and need to be considered accordingly.

**•** *Defined roles and responsibilities* for organizing quality assurance activities.

for defect detection in MDE environments.

184 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

for defect detection tool support.

defect detection processes.

models across disciplines.

ing processes:

the method/tool.

The application of SWT mechanisms requires knowledge engineering capabilities for data management, mapping, and querying. However, the scarce availability of knowledge engineers and the general scarcity of engineering resources [29] in the automation systems engineering (ASE) domain requires also "light-weight" defect detection approaches with and without SWT capabilities. *Light-weight* refers to the application of basic SWT concepts in engineering projects including support for defect detection and without high additional effort for implementing a comprehensive SWT solution for defect detection. **Figure 2** illustrates the *defect detection framework* with different levels of semantic integration approaches.

**Defect detection level 0: isolated engineering plans and heterogeneous data sources**. Current and traditional MDE approaches take as input heterogeneous artifact sources coming from various disciplines. This defect detection level does not include SWT contributions. Thus, defect detection processes are purely human based and require considerable manual effort for reviewing. Experts have to bridge the semantic gaps between engineering artifacts manually or by applying isolated support solutions that require high effort for application and maintenance [30]. This applies specifically in the case of data model changes. Beyond the effortconsuming manual activities, high cognitive effort is required by experts [31]. The complexity of engineering plans often hides defects very well in various parts of and views on engineering plans. Thus, in MDE, defect detection on this level is often ineffective and always inefficient. While defect detection can be improved by applying reading techniques, such as scenariobased reading or perspective-based reading [32], the high cognitive effort and the manual bridging of semantic gaps still limit defect detection effectiveness and efficiency.

**Defect detection level 1: common concepts bridge heterogeneous data models based on commonly used data aspects**. Wache *et al*. [33] distinguish between three ontology-based approaches:, *single-ontology, multiple-ontology*, and *hybrid approaches*, to achieve semantic integration, as follows:

**Figure 2.** Conceptual approach for the defect detection framework (DDF) in the context of SWT contributions on three levels.

*Single-ontology approaches* use one ontology as a semantic bridge to integrate several data sources. The challenge is to create an ontology that is sufficiently broad in its coverage to describe all data sources that are integrated. Additionally, an introduction of a new data source requires changes to the ontology.

*Multiple-ontology approaches* create an ontology to semantically describe each data source to be integrated (i.e., local ontologies) and then create pair-wise mappings between these ontologies. **•** *Hybrid approaches* on the other hand combine the previous strategies as follows: Local ontologies are created to describe each data source and are then mapped to a global ontology that contains concepts common to all integrated data sources (i.e., common concepts). The hybrid approach enables engineers to continue working in their well-known environments and enable data synchronization via these common concepts. For instance, in hydro power plant engineering, signals can be used as a common concept. A signal corresponds to electrical wiring or power levels of connectors (electrical discipline), physical layout and position of wires (mechanical discipline), and software variables for control software (software discipline). In the context of defect detection, common concepts enable reviewing engineering aspects by the review team, linked by common concepts. A major advantage of this approach is the lower cognitive burden of reviewers by providing clear links between engineering plans. Tools can be used to automate search tasks during the review by using querying mechanisms. However, this approach requires the upfront investment of defining and maintaining the common concepts data structures.

nance [30]. This applies specifically in the case of data model changes. Beyond the effortconsuming manual activities, high cognitive effort is required by experts [31]. The complexity of engineering plans often hides defects very well in various parts of and views on engineering plans. Thus, in MDE, defect detection on this level is often ineffective and always inefficient. While defect detection can be improved by applying reading techniques, such as scenariobased reading or perspective-based reading [32], the high cognitive effort and the manual

**Defect detection level 1: common concepts bridge heterogeneous data models based on commonly used data aspects**. Wache *et al*. [33] distinguish between three ontology-based approaches:, *single-ontology, multiple-ontology*, and *hybrid approaches*, to achieve semantic

**Figure 2.** Conceptual approach for the defect detection framework (DDF) in the context of SWT contributions on three

*Single-ontology approaches* use one ontology as a semantic bridge to integrate several data sources. The challenge is to create an ontology that is sufficiently broad in its coverage to describe all data sources that are integrated. Additionally, an introduction of a new data source

*Multiple-ontology approaches* create an ontology to semantically describe each data source to be integrated (i.e., local ontologies) and then create pair-wise mappings between these ontologies.

bridging of semantic gaps still limit defect detection effectiveness and efficiency.

186 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

integration, as follows:

levels.

requires changes to the ontology.

**Defect detection level 2: semantically integrated data**. All-in-one solutions (tool suites) typically consist of a common database including plans coming from all engineering disciplines [2] in a homogenous engineering environment. Yet, limitations of that approach include vendor lock-in, limited data exchange capabilities with tools that are not part of the tool suite (common in project consortia [34]), or low flexibility regarding the extensibility of the common data model.

In context of MDE for ASE environments, heterogeneous data models come from different sources and, as such, a solution that assumes a homogeneous data mode is not feasible. In such settings, SWTs can help to provide an integrated view on engineering data from various perspectives. SWT-based engineering model integration aims to bridge semantic gaps in engineering environments between project participants and their tools, who use different local terminologies [35]. Integrated data can then support the analysis, automation, and improvement in MDE processes. Semantic model integration is defined as solving problems originating from the intent to share information across disparate and semantically heterogeneous data [36]. These problems include the matching of data schemes, the detection of duplicate entries, the reconciliation of inconsistencies, and the modeling of complex relations in different data sources [37]. Integrated data and views on engineering data can support defect detection by (a) preparing review packages, that is, scoping specific parts of the overall system for reviewing purposes, and (b) enabling the detection of cross-disciplinary defects, thanks to the explicitly represented links between different disciplines established during the integration process.

While common concepts on level 1 represent manually stated links between data originating from different disciplines, links on level 2 are established semi-automatically by using explicit engineering knowledge. While the semantic integration with SWTs already supports defect detection activities, SWTs can be employed to create defect detection mechanisms, in particular, through semantic querying with SPARQL. Benefits of these approaches are: (a) defect detection tasks are explicitly represented in SPARQL queries, thus allowing periodic query repetition; (b) since queries typically address the ontology concepts, the checks remain valid even if the underlying data model, for example, of signal concepts, changes; and (c) semantic query languages rely on reasoning mechanisms and are, therefore, well fitted [38] to check model consistency and coherency.

## **5. Industry use cases and lessons learned**

This section presents two key *Use Cases* (UCs) developed in cooperation with industry partners and prototype solutions in context of the defect detection framework (DDF) without/with Semantic Web Technologies.

#### **5.1. Defect detection without Semantic Web Technologies (level 0)**

On Level 0, where a set of isolated engineering plans based on heterogeneous data sources is available, no SWTs are used. Engineering knowledge is implicitly embedded within domain experts [24], who are familiar with at least two engineering disciplines, for example, electrical and software or electrical and process discipline, as depicted in **Figure 2**. These experts bridge the semantic gap between aforementioned engineering disciplines manually. Thus, defect detection can become time-consuming and risky because experts can oversee defects easily within a huge amount of given data. For instance, in the hydro power plant domain, 40.000 data points (i.e., signals) are available and need to be reviewed/inspected by experts. Tool solutions like *Gerrit* can be used to compare different engineering model/plan versions to highlight changes to better drive the review process. However, the availability of a textual representation is required for enabling *Gerrit* applications. If no textual documents are available but engineering plans (such as diagrams) are present, *DefectRadar* can be used for annotation and review purposes [39].

Independent of applied tools, guidelines from Review/Inspection, such as perspective-based reading, scenario-based reading, or checklist-based reading technique approaches, can be used to focus on perspectives (defect detection from experts coming from specific disciplines) or scenarios (focus on use cases and application scenarios). Checklists based on the ISO 25010 standard [40] can represent a complementary approach to systematically check important (predefined and application-specific) items to support defect detection. Although such guidelines support review processes, the review itself is still performed manually and often paper based [41].

#### **5.2. Common concepts with limited Semantic Web Technologies (level 1)**

The first (basic) level of SWT contributions includes the application of common concepts as a hybrid ontology (see Section 4.2). The main idea of this approach is to elicit commonly used data from involved disciplines (even if they have different terminologies), transform them to a common concept, and map these common data between different disciplines. Thus, specifically used data are left within local tool solutions, and common concepts are centrally stored and are used for mapping data elements between related disciplines. Thus, changes in one discipline can be passed efficiently via common concepts to related disciplines by highlighting changes. Hence, the focus for review is based on deviations and changes (similar to the *Gerrit* approach). **Figure 3** presents a simple workflow for difference/defect detection approach, and **Figure 4** shows a screenshot of the prototype implementation (i.e., highlighting deviations/changes of different engineering model/plan version as input for review).

**Figure 3.** Basic workflow for difference/defect detection with focused reviews.

query languages rely on reasoning mechanisms and are, therefore, well fitted [38] to check

This section presents two key *Use Cases* (UCs) developed in cooperation with industry partners and prototype solutions in context of the defect detection framework (DDF) without/with

On Level 0, where a set of isolated engineering plans based on heterogeneous data sources is available, no SWTs are used. Engineering knowledge is implicitly embedded within domain experts [24], who are familiar with at least two engineering disciplines, for example, electrical and software or electrical and process discipline, as depicted in **Figure 2**. These experts bridge the semantic gap between aforementioned engineering disciplines manually. Thus, defect detection can become time-consuming and risky because experts can oversee defects easily within a huge amount of given data. For instance, in the hydro power plant domain, 40.000 data points (i.e., signals) are available and need to be reviewed/inspected by experts. Tool solutions like *Gerrit* can be used to compare different engineering model/plan versions to highlight changes to better drive the review process. However, the availability of a textual representation is required for enabling *Gerrit* applications. If no textual documents are available but engineering plans (such as diagrams) are present, *DefectRadar* can be used for

Independent of applied tools, guidelines from Review/Inspection, such as perspective-based reading, scenario-based reading, or checklist-based reading technique approaches, can be used to focus on perspectives (defect detection from experts coming from specific disciplines) or scenarios (focus on use cases and application scenarios). Checklists based on the ISO 25010 standard [40] can represent a complementary approach to systematically check important (predefined and application-specific) items to support defect detection. Although such guidelines support review processes, the review itself is still performed manually and often

The first (basic) level of SWT contributions includes the application of common concepts as a hybrid ontology (see Section 4.2). The main idea of this approach is to elicit commonly used data from involved disciplines (even if they have different terminologies), transform them to a common concept, and map these common data between different disciplines. Thus, specifically used data are left within local tool solutions, and common concepts are centrally stored and are used for mapping data elements between related disciplines. Thus, changes in one discipline can be passed efficiently via common concepts to related disciplines by highlighting changes. Hence, the focus for review is based on deviations and changes (similar to

**5.2. Common concepts with limited Semantic Web Technologies (level 1)**

model consistency and coherency.

Semantic Web Technologies.

annotation and review purposes [39].

paper based [41].

**5. Industry use cases and lessons learned**

188 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

**5.1. Defect detection without Semantic Web Technologies (level 0)**

This concept has been implemented and evaluated as *"Engineering Object Change Management"* (UC 1) in the hydro power plant domain [42]. In the context of this use case *signals* can be seen as the foundation for common concepts that enable efficient data exchange between engineering disciplines and synchronization of heterogeneous engineering plans. A *Virtual Common Data Model* (VCDM) has been elicited with industry experts to link individual disciplines [43, 44]. Note that key parts of signals (coming from different disciplines) have to be identified, transformed to the VCDM, and mapped to enable a seamless data integration. A database holds the VCDM data and provides links to discipline-specific tools and data models. Based on this VCDM concept, a change management process supported by a web-based tool has been designed at the industry partner.

**Figure 4.** Prototype screenshot of difference checks of the prototype implementation.

The prototype tool is capable of handling changes, that is, new, modified, or removed signals across disciplines and domain borders. A *difference view*, that is, a comparison of already available signals and modified signals, enabled engineers to assess the signal data and to decide whether the deviation should be accepted as a change or be classified as a defect. The main results of the pilot study [4] were that the prototype solution enabled (a) more effective and efficient data exchange in all phases of the engineering process and (b) observation capabilities of the change management process for project management and control.

In the context of defect detection, the explicit view on "deviation types" (i.e., changes or defects) provides the foundation for a more effective and efficient review process of signal data. An important benefit of this approach is the focus on the most critical deviations from a prior data version instead of reviewing the overall set of engineering data (signals) of the hydro power plant. Note that a typical hydro power plant includes a set of up to 40.000 signals and a small subset (2–5%) of deviations that allow to focus the review to 800 to 2.000 deviating signals. Although the prototype solution provides support for defect detection by presenting differences and deviations to the reviewer, the review process is executed by a human expert team without any specific guidance for defect detection. In this context, guidance refers to applying checklists, scenarios, or perspectives to identify defects more effective and efficient. Justification of defects typically relies on expert domain knowledge and engineering experience and is based on industry and organization best practices.

Although the automated identification of deviations and changes has been found useful by experts in context of the prototype implementation, additional guidelines, such as reading techniques, can help improve defect detection performance in terms of increased effectiveness (i.e., increased number of defects found) and efficiency (i.e., identifying more defects per time interval), and decreased false positives (i.e., wrongly reported defects).

Based on the provided focus on deviations and changes, the *"Focused Inspection"* [45] and "*Model Quality Assurance*" approaches [46] help to improve the unguided defect detection process by supporting (a) the review team selection and (b) their reading process. Individual engineers focus on defined aspects of the systems, for example, on mechanical, electrical, or software characteristics of the engineering plan, and their overlaps with partner disciplines. Thus, the reading technique can take into account individual disciplines, perspectives, and typical candidate defects that occur in discipline-specific engineering plans and across engineering plans. In this use case, *common concepts* (and the VCDM) are used as vehicle for data synchronization and for improving reviewing processes. The reviewing process is supported by a difference view (tool support) and reading technique (method support). In the pilot study [4], both the defect detection effectiveness and efficiency increased. However, an additional knowledge experts or key users that are familiar with knowledge engineering are required for supporting the common concept elicitation, transformation, and mapping process. In the prototype solution, a knowledge expert, that is, one of the authors, supports these process steps. However, additional tool support is planned as future work that guides domain experts and key users to conduct these knowledge management tasks without deep understanding of SWT mechanisms.

#### **5.3. Integrated data with advanced Semantic Web Technologies (level 2)**

The second (advanced) level of SWT contributions includes the establishment of integrated data based on a commonly used data exchange language (see Section 4.3). The *AutomationML* *Analyzer*<sup>9</sup> (UC 2) uses SWTs to provide an interface for analyzing data from integrated *AutomationML* files.

*AutomationML* [14] is an emerging data format for exchanging engineering data [47]. While the concerted use of *AutomationML* in an engineering project makes the integration between data from different engineering disciplines easier, tools are still needed for navigating and analyzing integrated *AutomationML* data more easily. To that end, the *AutomationML Analyzer* [48] uses ontology-based technologies to integrate *AutomationML* data to provide easy navigation support within the *AutomationML* data as well as to detect project level inconsistencies and defects through SPARQL querying of the integrated data. The main SWT capabilities used are (a) semantic modeling of an *AutomationML* ontology, which enables semantically enriching the input data; (b) browsing and exploration of the semantic data through *Linked Data*-based mechanisms; and (c) the use of reasoning mechanisms as part of the SPARQL querying activities.

This prototype fits Level 2 of the framework because it uses SWTs both for data integration and for defect detection activities. **Figure 5** presents the concept for the *AutomationML Analyzer*, including review process support. Core components of the concept include the *AutomationML Analyzer* component, providing integrated data and the *review process support* component that enables required review functionality such as browsing through the plant topology, automated execution of defined queries to identify deviations and changes, and querying capabilities for reporting. As a prerequisite, engineering data are available in *AutomationML* data format. However, the concept is applicable to any structured data formats required within a project. In the context of this prototype, the *AML.hub* [17, 49] represents the platform for data synchronization (i.e., mapping and merging of data coming from heterogeneous sources).

**Figure 5.** *AML Analyzer* concept for review support [48].

efficient data exchange in all phases of the engineering process and (b) observation capabilities

In the context of defect detection, the explicit view on "deviation types" (i.e., changes or defects) provides the foundation for a more effective and efficient review process of signal data. An important benefit of this approach is the focus on the most critical deviations from a prior data version instead of reviewing the overall set of engineering data (signals) of the hydro power plant. Note that a typical hydro power plant includes a set of up to 40.000 signals and a small subset (2–5%) of deviations that allow to focus the review to 800 to 2.000 deviating signals. Although the prototype solution provides support for defect detection by presenting differences and deviations to the reviewer, the review process is executed by a human expert team without any specific guidance for defect detection. In this context, guidance refers to applying checklists, scenarios, or perspectives to identify defects more effective and efficient. Justification of defects typically relies on expert domain knowledge and engineering experi-

Although the automated identification of deviations and changes has been found useful by experts in context of the prototype implementation, additional guidelines, such as reading techniques, can help improve defect detection performance in terms of increased effectiveness (i.e., increased number of defects found) and efficiency (i.e., identifying more defects per time

Based on the provided focus on deviations and changes, the *"Focused Inspection"* [45] and "*Model Quality Assurance*" approaches [46] help to improve the unguided defect detection process by supporting (a) the review team selection and (b) their reading process. Individual engineers focus on defined aspects of the systems, for example, on mechanical, electrical, or software characteristics of the engineering plan, and their overlaps with partner disciplines. Thus, the reading technique can take into account individual disciplines, perspectives, and typical candidate defects that occur in discipline-specific engineering plans and across engineering plans. In this use case, *common concepts* (and the VCDM) are used as vehicle for data synchronization and for improving reviewing processes. The reviewing process is supported by a difference view (tool support) and reading technique (method support). In the pilot study [4], both the defect detection effectiveness and efficiency increased. However, an additional knowledge experts or key users that are familiar with knowledge engineering are required for supporting the common concept elicitation, transformation, and mapping process. In the prototype solution, a knowledge expert, that is, one of the authors, supports these process steps. However, additional tool support is planned as future work that guides domain experts and key users to conduct these knowledge management tasks without deep

of the change management process for project management and control.

190 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

ence and is based on industry and organization best practices.

understanding of SWT mechanisms.

interval), and decreased false positives (i.e., wrongly reported defects).

**5.3. Integrated data with advanced Semantic Web Technologies (level 2)**

The second (advanced) level of SWT contributions includes the establishment of integrated data based on a commonly used data exchange language (see Section 4.3). The *AutomationML* For establishing the *AutomationML Analyzer*(and the underlying mechanisms), a knowledge engineer is required to perform a multitude of tasks, such as transformations, data integration,

<sup>9</sup> AutomationML Analyzer: http://data.ifs.tuwien.ac.at/aml/analyzer, access 2016-09.

and querying capabilities. However, in common industry projects where similar projects are conducted, this effort represents initial activities and the results, for example, transformation, mapping strategies, and queries, can be reused for similar projects within the organization.

#### **6. Discussion, limitations, and future work**

The main goal of this paper focuses on building a platform for users and domain experts, that is, experts in MDE and ASE environments, for executing review activities more effectively and efficiently (process improvement) and to improve *defect detection* performance in terms of increased defect detection effectiveness and efficiency (method application improvement). Several strategies can help to improve defect detection processes and activities by using a certain degree of Semantic Web Technology involvement. The defect detection framework (DDF) provides a basic framework for approaches in context of defect detection with a different degree of SWT contributions.

**Table 1** presents *identified requirements and expected key capabilities* (y-axis) for tool solutions (RI.1). These requirements and capabilities have been elicited in collaboration with industry and research experts (Section 4.1). RI.2 focuses on the development a *defect detection framework (DDF)*, illustrated in **Figure 2** and represented by the x-axis in **Table 1**, which is capable of providing an overview of different levels of SWT contributions for defect detection. Within the framework of this chapter, three levels of SWT contributions have been found useful for assessing the level of SWT contribution and for driving improvement initiatives in context of quality assurance and defect detection. On different level of SWT contributions, selected industry real-life prototypes have been developed and conceptually evaluated (RI.3) to elicit benefits and limitations of individual SWT-related solutions. Domain experts bridge technical gaps between isolated tools and semantic gaps between heterogeneous data models manually with limited tool support. In addition, defect detection is executed manually with high cognitive effort. To overcome these limitations (i.e., high effort and error-proneness of manual defect detection), common concepts help to bridge the gap based on commonly available data. *Defect Detection Level 1* focuses on these common concepts, that is, commonly used data that support data exchange between disciplines. Transformations and mappings of local tool data to common concepts help to bridge the semantic gaps of individual data. Based on these mappings, changes can be propagated via common concepts from one discipline to the other. Furthermore, defect detection is supported by providing difference/deviation checks to focus on differences between artifact versions or artifacts derived from different disciplines (see **Figure 4** in Section 5.2). This approach has been successfully implemented in industry (see Section 5.2) which supports the engineering team *Defect Detection Level 2*, uses advanced approaches from SWTs, and builds on semantically integrated data that enable browsing, querying, and automation-supported defect detection and reporting—implemented in the *AutomationML Analyzer* (see Section 5.3). Note that this approach uses *AutomationML*, an emerging standard for data exchange in MDE environments for enabling data management, that is, data mapping and synchronization. Yet, the conceptual approach supports various types of data formats and has been implemented with *AutomationML* data (see Section 5.3).


**Table 1.** Conceptual evaluation of the defect detection framework.

and querying capabilities. However, in common industry projects where similar projects are conducted, this effort represents initial activities and the results, for example, transformation, mapping strategies, and queries, can be reused for similar projects within the organization.

The main goal of this paper focuses on building a platform for users and domain experts, that is, experts in MDE and ASE environments, for executing review activities more effectively and efficiently (process improvement) and to improve *defect detection* performance in terms of increased defect detection effectiveness and efficiency (method application improvement). Several strategies can help to improve defect detection processes and activities by using a certain degree of Semantic Web Technology involvement. The defect detection framework (DDF) provides a basic framework for approaches in context of defect detection with a different

**Table 1** presents *identified requirements and expected key capabilities* (y-axis) for tool solutions (RI.1). These requirements and capabilities have been elicited in collaboration with industry and research experts (Section 4.1). RI.2 focuses on the development a *defect detection framework (DDF)*, illustrated in **Figure 2** and represented by the x-axis in **Table 1**, which is capable of providing an overview of different levels of SWT contributions for defect detection. Within the framework of this chapter, three levels of SWT contributions have been found useful for assessing the level of SWT contribution and for driving improvement initiatives in context of quality assurance and defect detection. On different level of SWT contributions, selected industry real-life prototypes have been developed and conceptually evaluated (RI.3) to elicit benefits and limitations of individual SWT-related solutions. Domain experts bridge technical gaps between isolated tools and semantic gaps between heterogeneous data models manually with limited tool support. In addition, defect detection is executed manually with high cognitive effort. To overcome these limitations (i.e., high effort and error-proneness of manual defect detection), common concepts help to bridge the gap based on commonly available data. *Defect Detection Level 1* focuses on these common concepts, that is, commonly used data that support data exchange between disciplines. Transformations and mappings of local tool data to common concepts help to bridge the semantic gaps of individual data. Based on these mappings, changes can be propagated via common concepts from one discipline to the other. Furthermore, defect detection is supported by providing difference/deviation checks to focus on differences between artifact versions or artifacts derived from different disciplines (see **Figure 4** in Section 5.2). This approach has been successfully implemented in industry (see Section 5.2) which supports the engineering team *Defect Detection Level 2*, uses advanced approaches from SWTs, and builds on semantically integrated data that enable browsing, querying, and automation-supported defect detection and reporting—implemented in the *AutomationML Analyzer* (see Section 5.3). Note that this approach uses *AutomationML*, an emerging standard for data exchange in MDE environments for enabling data management, that is, data mapping and synchronization. Yet, the conceptual approach supports various types of data formats and has been implemented with *AutomationML* data (see Section 5.3).

**6. Discussion, limitations, and future work**

192 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

degree of SWT contributions.

However, to enable the elicitation of common concepts, transformation and mapping (for Level 1) and the construction of integrated data (for Level 2) requires knowledge engineers who support users and domain experts, elicit, and implement SWT approaches.

**Table 1** presents an assessment of SWT contribution levels (and prototype solutions) and identified requirements for process and tool support in context of defect detection. *Humanbased defect detection* (Level 0) is based on expert activities and their expertise with limited process support and limited tool support. However, due to organizational activities, reviews and inspection can still be implemented in MDE environments on a basic level (e.g., by using organizational guidelines and supporting tools). *Common Concepts* (Level 1) support defect detection by enabling links and mechanisms for data synchronization and defect detection, for example, highlighting changes in artifacts derived from different disciplines or artifact versions, to focus on recent changes and deviations. Instead of reviewing thousands of engineering data, they can focus on deviations, i.e., changes, inconsistencies, or defects. In the evaluation context, the prototypes showed:


**•** some basic tool support.

*Integrated Data* (Level 2) builds on a high degree of SWT and data integration and enables advanced defect detection by also supporting review processes, querying, navigation, and enhanced tool support.

With focus on the *defect detection framework* (RI.2), the three-level concept has considerable benefits for classifying SWT contributions as starting point for improving/introducing SWTs in an organization to improve collaboration and defect detection performance. On the other hand, defect detection performance can also be improved with low effort, by introducing systematic reading techniques or tool support to focus on most critical and important deviations. Nonetheless, to benefit from SWTs capabilities, for example, queries, reporting, and navigation, knowledge engineering capabilities are required to support engineers in data management and query definition.

#### **6.1. Limitations and threats to validity**

The *Requirements and Tool Capabilities* have been detailed and discussed with representative industry and research experts in the field of automation systems engineering. These industry and research experts have been recruited from the automation systems development domains, for example, the hydro power plant domain. Although identified requirements and capabilities match to the needs of these experts, additional requirements and needed tool capabilities might arise if different application domains are considered, for example, building automation, factory automation, or chemical production systems. For instance, common concepts need to be adapted according to available data in related engineering domains.

The *defect detection framework (DDF)* has been developed within the context of the current research work. It has been executed in the CDL-Flex, which is driven by industry partners and their respective individual needs. However, it is the view of the authors that the framework may assist industry and research entities so as to drive and foster improvement initiatives. Although this basic framework has been found useful in the evaluation context, additional levels of SWT contributions might be reasonable and applicable for different application domains. Further research work is needed to include additional applied SWT contributions applicable in different domains.

Selected *Use Cases and Prototype Solutions* have been developed based on industry-specific requirements that have been derived from industry partners in the automation systems domain, mainly hydro power plant engineering, steel mill engineering, or production automation engineering. Assuming comparable requirements in different domains, SWT contributions need to be evaluated in these additional application domains. Nonetheless, the conceptual real-life use cases and prototype implementations have been evaluated in various industry contexts successfully (see Section 4 for the solution concepts and Section 5 for individual industry real-life user cases). Based on described prototype solutions, additional application experiences in the defined industry context (i.e., business domain of industry partners) are needed. In addition, requirements from related business domains need to be explored and considered in the presented solution concepts. Therefore, the plans include more detailed large-scale empirical studies (e.g., case studies) in real-world industry settings at existing industry partners and beyond. SWTs can make defect detection more effective and efficient. It is worth noting though that a higher level of SWT contribution requires considerable knowledge engineering capabilities to support data management and query definition. Although SWT concepts have been successfully applied in industry, their application can be risky without good understanding of their success factors and limitations.

#### **6.2. Future work**

**•** some basic tool support.

194 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

enhanced tool support.

management and query definition.

applicable in different domains.

**6.1. Limitations and threats to validity**

*Integrated Data* (Level 2) builds on a high degree of SWT and data integration and enables advanced defect detection by also supporting review processes, querying, navigation, and

With focus on the *defect detection framework* (RI.2), the three-level concept has considerable benefits for classifying SWT contributions as starting point for improving/introducing SWTs in an organization to improve collaboration and defect detection performance. On the other hand, defect detection performance can also be improved with low effort, by introducing systematic reading techniques or tool support to focus on most critical and important deviations. Nonetheless, to benefit from SWTs capabilities, for example, queries, reporting, and navigation, knowledge engineering capabilities are required to support engineers in data

The *Requirements and Tool Capabilities* have been detailed and discussed with representative industry and research experts in the field of automation systems engineering. These industry and research experts have been recruited from the automation systems development domains, for example, the hydro power plant domain. Although identified requirements and capabilities match to the needs of these experts, additional requirements and needed tool capabilities might arise if different application domains are considered, for example, building automation, factory automation, or chemical production systems. For instance, common concepts need to

The *defect detection framework (DDF)* has been developed within the context of the current research work. It has been executed in the CDL-Flex, which is driven by industry partners and their respective individual needs. However, it is the view of the authors that the framework may assist industry and research entities so as to drive and foster improvement initiatives. Although this basic framework has been found useful in the evaluation context, additional levels of SWT contributions might be reasonable and applicable for different application domains. Further research work is needed to include additional applied SWT contributions

Selected *Use Cases and Prototype Solutions* have been developed based on industry-specific requirements that have been derived from industry partners in the automation systems domain, mainly hydro power plant engineering, steel mill engineering, or production automation engineering. Assuming comparable requirements in different domains, SWT contributions need to be evaluated in these additional application domains. Nonetheless, the conceptual real-life use cases and prototype implementations have been evaluated in various industry contexts successfully (see Section 4 for the solution concepts and Section 5 for individual industry real-life user cases). Based on described prototype solutions, additional application experiences in the defined industry context (i.e., business domain of industry partners) are needed. In addition, requirements from related business domains need to be explored and considered in the presented solution concepts. Therefore, the plans include more

be adapted according to available data in related engineering domains.

Future work focuses on evaluating and improving the presented solution concepts and prototype solutions, extending and enhancing the prototype solution within the current application domains and addressing needs from additional application domains, and extending the platform toward a comprehensive process support for enabling and improving comprehensive reviewing with SWT and tool support. More specifically, future work includes:


It is the authors' view that SWTs can make defect detection more effective and efficient. It is worth noting though that a higher level of SWT contribution requires considerable knowledge engineering capabilities to support data management and query definition. Although SWT concepts have been successfully applied in industry, their application can be risky without good understanding of their success factors and limitations [50].

## **Acknowledgements**

Part of this work was supported by the Christian Doppler Forschungsgesellschaft, the Federal Ministry of Economy, Family and Youth and the National Foundation for Research, Technology and Development, Austria.

## **Author details**

Dietmar Winkler1,2\*, Marta Sabou2 and Stefan Biffl2

\*Address all correspondence to: dwinkler@sba-research.org

1 SBA Research gGmbH & Institute of Software Technology and Interactive Systems, CDL-Flex, Vienna University of Technology, Vienna, Austria

2 Institute of Software Technology and Interactive Systems, CDL-Flex, Vienna University of Technology, Vienna, Austria

## **References**


[8] Biffl S., Sabou M., editors. Semantic Web for Intelligent Engineering Applications. 1st ed. Springer, New York; 2016.

**Author details**

Dietmar Winkler1,2\*, Marta Sabou2

Technology, Vienna, Austria

(INCOM); 2015.

**References**

and Stefan Biffl2

1 SBA Research gGmbH & Institute of Software Technology and Interactive Systems, CDL-

2 Institute of Software Technology and Interactive Systems, CDL-Flex, Vienna University of

[1] Biffl S., Schatten A., Zoitl A. Integration of heterogeneous engineering environments for the automation systems lifecycle. In: Proceedings of the 7th International Conference on Industrial Informatics (INDIN); 23–26 June 2009; Cardiff, Wales. IEEE; 2009. p.

[2] Fay A., Biffl S., Winkler D., Drath R., Barth M. A Method to evaluate the openness of automation tools for increased interoperability. In: Proceedings of the 39th Annual Conference of the IEEE Industrial Electronics Society (IECON); 10–13 November; IEEE;

[3] Biffl S., Lüder A., Winkler D. Multi-disciplinary engineering for Industrie 4.0: semantic challenges, needs, and capabilities. In: Biffl S., Sabou M., editors. Semantic Web for

[4] Winkler D., Moser T., Mordinyi R., Sunindyo W.D., Biffl S. Engineering object change management process observation in distributed automation systems projects. In: Industrial Proceedings of the 18th EuroSPI Conference; Roskilde, Denmark; 2013.

[5] Feldmann S., Herzig S.J.I., Kernschmidt K., Wolfenstetter T., Kammerl D., Qamar A., Lindemann U., Krcmar H., Paredis C.J.J., Vogel-Heuser B. Towards effective management of inconsistencies in model-based engineering of automated production systems. In: Proceedings of the IFAC Symposium on Information Control in Manufacturing

[6] Rössler P. Schlich M., Kneuper R. Reviews in der System- und Softwareentwicklung: Grundlagen, Praxis, kontinuierliche Verbesserung. 1st ed. dpunkt.verlag; 213. 176 p.

[7] Kovalenko O., Serral E., Sabou M., Ekaputra F.J., Winkler D., Biffl S. Automating crossdisciplinary defect detection in multi-disciplinary engineering environments. In: Proceedings of 19th International Conference on Knowledge Engineering and Knowl-

edge Management (EKAW); Linköping, Sweden. 2014.

Intelligent Engineering Applications. 1st ed. Springer, New York; 2016.

\*Address all correspondence to: dwinkler@sba-research.org

Flex, Vienna University of Technology, Vienna, Austria

196 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

576–581. doi: 10.1109/INDIN.2009.5195867

2013. doi: 10.1109/IECON.2013.6700266


[21] Lanubile F., Mallardo T., Calefato F. Tool support for geographically dispersed inspection teams. Software Process Improvement and Practice. 2003;8(4):217–231.

[22] Anderson P., Reps T., Teitelbaum T., Zarins M. Tool support for fine-grained software

[23] Winkler D., Biffl S. Focused inspections to support defect detection in multi-disciplinary engineering environments. In: Proceedings of the 16th International Conference on Product-Focused Software Process Improvement (PROFES); Bozen-Bolzano, Italy.

[24] Winkler D., Musil J., Musil A., Biffl S. Collective intelligence-based quality assurance: combining inspection and risk assessment to support process improvement in multidisciplinary engineering. In: Proceedings of the 23rd EuroSPI Conference;

[25] Feldmann S., Herzig S.J., Kernschmidt K.,Wolfenstetter T., Kammerl D., Qamar A., Lindemann U., Krcmar H., Paredis C., Vogel-Heuser B. A comparison of inconsistency management approaches using a mechatronic manufacturing system design case study. In: Proceedings of the IEEE International Conference on Automation Science and

[26] Kovalenko O., Moser T. Using explicit and machine-understandable engineering knowledge for defect detection in automation systems engineering. In: Proceedings of International Doctoral Symposium on Software Engineering and Advanced Applica-

[27] Abele L., Legat C., Grimm S., Müller A.W. Ontology-based validation of plant models. In: 11th IEEE International Conference on Industrial Informatics (INDIN); IEEE; 2013.

[28] Feldmann S., Rösch S., Legat C., Vogel-Heuser B. keeping requirements and test cases consistent: towards an ontology-based approach. In: 12th IEEE International Confer-

[29] Ekaputra F., Serral E., Winkler D., Biffl S. An analysis framework for ontology querying tools. In: Proceedings of the 9th International Conference on Semantic Systems (I-SEMANTICS) in conjunction with the 13th International Conference on Knowledge

[30] Biffl S., Mordinyi R., Moser T. Anforderungsanalyse für das integrierte Engineering – Mechanismen und Bedarfe aus der Praxis. ATP Edition; 2012;54(5):28–35.

[31] Blackwell A., Green T. Notational systems—the cognitive dimensions of notations framework. In: Carroll J., editor. HCI Models, Theories, and Frameworks: Toward a

Multidisciplinary Science.Morgan Kaufmann, San Francisco; 2003. 103–134 p.

Management and Knowledge Technologies (I-KNOW). ACM; 2013.

ence on Industrial Informatics (INDIN); 2014. 726–732 p.

14.-16.09.2016; Graz, Austria. Springer, New York; 2016. 163–175 p.

inspection. IEEE Software. 2003;20(4):42–50.

198 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Springer, New York; 2015.

Engineering (CASE). IEEE; 2015.

236–241 p.

tions (IDoSEAA); Oulu, Finland; 2011.


#### **The Use of Control Charts in the Study of Bitcoin's Price Variability** The Use of Control Charts in the Study of Bitcoin's Price Variability

#### Beata Szetela Beata Szetela

[46] Winkler D., Wimmer M., Biffl S. Model quality assurance for multi-disciplinary engineering. In: Biffl S., Lüder A., Gerhard D. (eds): Multi-Disciplinary Engineering of Cyber-Physical Production Systems, Book Chapter, Chapter 17, 2017 (upcoming). [47] Drath R., editor. Datenaustausch in der Anlagenplanung mit AutomationML: Integra-

[48] Sabou M., Ekaputra F.J., Kovalenko O. Supporting the engineering of cyber-physical production systems with the AutomationML analyzer. In: Proceedings of the CPPS

[49] Mordinyi R., Wimmer M., Biffl S. Versioning in multi-disciplinary engineering with the AutomationML Hub. 4th AutomationML User Conference, Esslingen, Germany, 2016

[50] Steyskal S., Wimmer M. Leveraging semantic web technologies for consistency management in multi-viewpoint systems engineering. In: Biffl S., Sabou M., editors. Semantic Web for Intelligent Engineering Applications. 1st ed. Springer, New York;

tion von CAEX, PLCopen XML und COLLADA. 1st ed. Springer; 2010.

Workshop; Vienna, Austria. 2016.

200 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

(upcoming).

2016.

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/66360

#### Abstract

The focus of this research is bitcoin's variability and its comparison with the variability of the EURO/USD exchange rate. Virtual currencies have been evolving in a dynamic way in the last few years. Under 600 different virtual currencies, the most successful was bitcoin. Its adherents saw in it an alternative to the traditional means of payments allowing the performance of real-time transactions at low costs. The accessibility, where no financial infrastructure is ensured or where either limited or no international agreements exist between financial and banking institutions was also an advantage. The opponents perceived this as a temporary curiosity with no future. Time confirmed that bitcoin has gained on popularity and the exchange rate to the main currencies rose in a dynamic way. The analysts, however, underline that the bitcoin is too volatile and unpredictable, so it cannot compete against the main currencies. The aim of this research is to compare the bitcoin (BTC) to US Dollar (USD) exchange rate and Euro to USD exchange rate volatility using control charts. The results have shown that BTC/USD exchange rate volatility is strongly affected by unexpected price jumps during the period (2010–2016), an act that significantly distinguishes it from more stable and predictable EUR/USD exchange rate variability.

Keywords: bitcoin, virtual currency, control chart, volatility, exchange rate, BTC/USD, EUR/USD

## 1. Introduction

Bitcoin is a virtual currency and a quite new phenomenon. It was created in 2008 by Satoshi Nakamoto, who published an article "Bitcoin: A Peer-to-Peer Electronic Cash System" in 2008, in which he described a concept of virtual, decentralized and independent means of payment, which is based on a cryptographic blockchain protocol [1]. His main idea was to build a currency based not on trust but on an algorithm, which cannot be influenced or manipulated. It was thought to be independent of any legal or governmental body. After the first release of

Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons

distribution, and eproduction in any medium, provided the original work is properly cited.

bitcoin, by generating the 'block genesis' in 2009, this currency was gaining rapidly in popularity.

As in September 2015 ca. 667 crypto currencies were established.1 Among them, bitcoin is considered to be the most popular and the most widely used. According to data published on bitinfocharts.com, bitcoin has the highest market capitalization of more than \$7 billion, representing 89% of the total capitalization of all cryptocurrency. Further down are the Ethereum \$574 million (7.1%), litecoin \$179 million (2.2%) and Dash \$39 million (0.5%). Data published by blockchain.info at the beginning of 2012 indicated that bitcoin had ca. 400 users. This number increased by ca. 970 k at the beginning of 2014 and reached ca. 8.5 m in September 2016 [2].

Within 4 years since its creation, the European Central Bank (ECB) and other financial institutions still have not come to a final conclusion, as to the classification of bitcoin and other cryptocurrencies. They have not recognized virtual currency as money or as a commodity. As such, the current legislation does not regulate events resulting thereof. The official definition of a 'virtual currency' was set up by a European Central Bank in 2012 for the first time, according to which 'a virtual currency is a type of unregulated, digital money, which is issued and usually controlled by its developers and used and accepted among the members of a specific virtual community' [3]. This definition associated virtual currency with a virtual world in a strict sense and the connection to the real economy was nearly neglected. Nowadays, bitcoin is similar to other virtual currencies that can be traded or exchanged for real money and goods and is accepted by many merchants all over the world. That is why in 2016, the International Monetary Fund (IMF) has extended the definition specifying, that 'virtual currencies can be obtained, stored, accessed and transacted electronically and can be used for a variety of purposes, as long as the transacting parties agree to use them' [4].

According to the IMF, the impact of virtual currencies on the real economy and the financial system is limited. It is, however, possible that with the increase of trading volume and acceptability, virtual currencies can become a serious threat to the financial and banking sector. One of the obstacles to the development of bitcoin named by IMF is its unstable variability.

The purpose of this study is to assess the validity of whether the dollar (USD) is more stable than the bitcoin (BTC) using xbar-s and CUSUM control charts. Depending on the outcome thereof, it will evaluate the concerns of sceptics pertaining to bitcoin and its further development and acceptance in the long term. If it turns out that the volatility of bitcoin does not deviate significantly from the volatility of dollar, then the fears of some financial institutions might appear to be valid.

## 2. Literature review

Bitcoin is gaining much more popularity not only among financiers but also among scientists. However, a limited number of scientific papers pertaining to cryptocurrency have been

<sup>1</sup> https://bitinfocharts.com/pl/index\_v.html

published. Given the above, the proposed content will be an important contribution to studies of both control cards and bitcoin. Current scientific achievements can be divided into four main fields of interest.


Generalized autoregressive conditional heteroscedasticity models (GARCHs) were applied to investigate similarities between bitcoin and both US dollar and gold [27, 28]. It was found that bitcoin-like other cryptocurrency tend to generate bubbles and that they do not have fundamental value [29]. Moreover, bitcoin fluctuations are characterized by sudden jumps and extreme pricing, which is characteristic for immature markets [30]. Autoregressive moving average and log-periodic power law models were applied to show that the price of bitcoin depends on the Chicago Board Options Exchange Index Volatility Index, which is indicative of speculation potential [31]. Other scientists argue that fluctuations in the price of Bitcoin are positively correlated with the amount of BTC users and are determined by the shocks of unknown sources of origin. The latter have an endogenous character and are not generated by the impact of specific variables, such as indexes S&P 500, gold rate against the US dollar or (XAU) and the Shanghai stock exchange index (SSE) [32]. According to Bouoiyour et al. [33], bitcoin's price fluctuations are best characterized by a generalized hyperbolic distribution.

The variability of bitcoin against the dollar in 2015 significantly decreased compared with the preceding period. The authors also claimed that bitcoin can be characterized by excessive asymmetry and the price is prone to the negative shocks negative than positive once.

## 3. Methodology

bitcoin, by generating the 'block genesis' in 2009, this currency was gaining rapidly in popu-

202 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

As in September 2015 ca. 667 crypto currencies were established.1 Among them, bitcoin is considered to be the most popular and the most widely used. According to data published on bitinfocharts.com, bitcoin has the highest market capitalization of more than \$7 billion, representing 89% of the total capitalization of all cryptocurrency. Further down are the Ethereum \$574 million (7.1%), litecoin \$179 million (2.2%) and Dash \$39 million (0.5%). Data published by blockchain.info at the beginning of 2012 indicated that bitcoin had ca. 400 users. This number increased by ca. 970 k at the beginning of 2014 and reached ca. 8.5 m in September 2016 [2].

Within 4 years since its creation, the European Central Bank (ECB) and other financial institutions still have not come to a final conclusion, as to the classification of bitcoin and other cryptocurrencies. They have not recognized virtual currency as money or as a commodity. As such, the current legislation does not regulate events resulting thereof. The official definition of a 'virtual currency' was set up by a European Central Bank in 2012 for the first time, according to which 'a virtual currency is a type of unregulated, digital money, which is issued and usually controlled by its developers and used and accepted among the members of a specific virtual community' [3]. This definition associated virtual currency with a virtual world in a strict sense and the connection to the real economy was nearly neglected. Nowadays, bitcoin is similar to other virtual currencies that can be traded or exchanged for real money and goods and is accepted by many merchants all over the world. That is why in 2016, the International Monetary Fund (IMF) has extended the definition specifying, that 'virtual currencies can be obtained, stored, accessed and transacted electronically and can be used for a variety of

According to the IMF, the impact of virtual currencies on the real economy and the financial system is limited. It is, however, possible that with the increase of trading volume and acceptability, virtual currencies can become a serious threat to the financial and banking sector. One

The purpose of this study is to assess the validity of whether the dollar (USD) is more stable than the bitcoin (BTC) using xbar-s and CUSUM control charts. Depending on the outcome thereof, it will evaluate the concerns of sceptics pertaining to bitcoin and its further development and acceptance in the long term. If it turns out that the volatility of bitcoin does not deviate significantly from the volatility of dollar, then the fears of some financial institutions

Bitcoin is gaining much more popularity not only among financiers but also among scientists. However, a limited number of scientific papers pertaining to cryptocurrency have been

of the obstacles to the development of bitcoin named by IMF is its unstable variability.

purposes, as long as the transacting parties agree to use them' [4].

might appear to be valid.

2. Literature review

https://bitinfocharts.com/pl/index\_v.html

1

larity.

Statistical process control (SPC) has found its application in many scientific areas. One of the tools, which are used by the SPC, is control charts (see Figure 1). A control chart (CC) is a graphical representation of a process. It presents an average value of the quality characteristics reflected on the chart by a central line (CL). Auxiliary lines, called upper control limit (UCL) and lower control limit (LCL), are used for the presentation of deviations from the mean of the process. Control limits are usually set as three times the standard deviation (3-sigma (σ) limits).

Figure 1. Control chart scheme.

The main idea behind control charts is to monitor an underlying process. If the observations fluctuate in a natural way within set advance control limits and if they do not reveal any specific patterns, then it is said that the process is under control. If, however, the monitored process breaks the established control limits, then it is understood that the process is out of control and specific actions should be launched to return the process under control. For further details considering control charts, please refer to Ref. [34].

SPC uses different types of charts, depending on the type of data used. For the continuous data, the following charts dedicated to variables are used:


For data based on countcharts for attributes are applied, e.g. np chart, p chart, c chart, u chart, etc.

For the purpose of this research, the following control charts will be applied, namely an xbar-s chart and a CUSUM with the moving range chart.

#### 3.1. Xbar-s chart

Xbar-s charts are used to monitor the variation and mean of the process. If the sample size (n) is not constant and is relatively large (n > 10), in such a case xbar-s charts are preferable against xbar-R chart.

For the unknown parameters of the s-bar chart and the variable sample size, the central line is defined as an average standard deviation for all samples (Eq. (1))

The Use of Control Charts in the Study of Bitcoin's Price Variability http://dx.doi.org/10.5772/66360 205

$$\overline{s} = \sqrt{\frac{\sum\_{i=1}^{m} (n\_i - 1)s\_i^2}{\sum\_{i=1}^{m} n\_i - m}} \tag{1}$$

where m is the number of samples, n<sup>i</sup> is the individual sample size and s<sup>i</sup> is an individual value of standard deviation for each sample defined as s ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ∑n i¼1 <sup>ð</sup>xi<sup>−</sup>x<sup>Þ</sup> 2 n−1 r .

The upper (Eq. (2)) and lower control limit (Eq. (3)), which define the boundaries for the (3σ) three-sigma control limits are calculated based on the following formulae:

$$\text{UCL} = \left(1 + \frac{3}{c\_4}\sqrt{1 - c\_4^2}\right)\overline{s} \tag{2}$$

$$LCL = \left(1 - \frac{3}{c\_4}\sqrt{1 - c\_4^2}\right)\overline{s} \tag{3}$$

where c<sup>4</sup> is a constant.

The main idea behind control charts is to monitor an underlying process. If the observations fluctuate in a natural way within set advance control limits and if they do not reveal any specific patterns, then it is said that the process is under control. If, however, the monitored process breaks the established control limits, then it is understood that the process is out of control and specific actions should be launched to return the process under control. For further details

SPC uses different types of charts, depending on the type of data used. For the continuous

For data based on countcharts for attributes are applied, e.g. np chart, p chart, c chart, u chart,

For the purpose of this research, the following control charts will be applied, namely an xbar-s

Xbar-s charts are used to monitor the variation and mean of the process. If the sample size (n) is not constant and is relatively large (n > 10), in such a case xbar-s charts are preferable against

For the unknown parameters of the s-bar chart and the variable sample size, the central line is

considering control charts, please refer to Ref. [34].

204 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

chart and a CUSUM with the moving range chart.


Figure 1. Control chart scheme.

etc.

3.1. Xbar-s chart

xbar-R chart.

data, the following charts dedicated to variables are used:


defined as an average standard deviation for all samples (Eq. (1))


Similarly, the control limits are determined for the xbar chart, which accompanies the schart. The central line is calculated as an average value of the individual averages (Eq. (4))

$$\text{CL} = \overline{\overline{\mathfrak{X}}} = \frac{\sum\_{i=1}^{m} n\_i \overline{\mathfrak{X}}\_i}{\sum\_{i=1}^{m} n\_i} \tag{4}$$

The three-sigma control limit is determined by the upper control limit (Eq. (5)) and the lower control limit (Eq. (6)) in the form:

$$\text{LCL} = \overline{\overline{\mathfrak{x}}} + \frac{\mathfrak{Z}\overline{\mathfrak{s}}}{c\_4\sqrt{n}}\tag{5}$$

$$UCL = \overline{\overline{x}} - \frac{3\overline{s}}{c\_4\sqrt{n}}\tag{6}$$

It is assumed that the underlying process is under control if it varies between defined control limits. The breach of any of the control limits points at the process being out of control.

#### 3.2. CUSUM and MR chart

Moving range chart enables to plot the sum of ranges of the adjacent pairs of observation within the investigated period. The central line (see Eq. (7)) is calculated as an average range of k-samples and the control limits (see Eqs. (8) and (9)) are set as m-times deviation from the average process range.

$$\overline{R} = \frac{\sum\_{i=1}^{k} R\_i}{k} \tag{7}$$

$$L\mathbb{C}\mathbb{L} = \overline{R}\text{-}md\_3\hat{\sigma} \tag{8}$$

$$L\mathbb{C}L = \overline{R} + md\_3\hat{\sigma} \tag{9}$$

where Ri is a range in sample i, k is the number of samples, m is a multiplier chosen to establish control limits, usually set to 3 and d<sup>3</sup> is a constant and σ^ is an estimated variance of a process.

CUSUM control charts plot the cumulative sum of deviation from the assumed target value (see Eq. (10)).

$$\mathbf{C}\_{i} = \sum\_{j=1}^{i} (\overline{\mathbf{x}}\_{j} - \mu\_{0}) \tag{10}$$

where Ci is a cumulative sum, xj is mean of a process in sample j, average and μ<sup>0</sup> is target value.

If the process is under control, then a cumulative sum (Ci) follows a random walk process with mean equals 0 [35]. It is assumed that the process is out of control, if the average values drift from the target value. If the values move in the positive direction, then the upper cumulative sum is written as in Eq. (11).

If, however, they move into the negative values, then the lower cumulative sum is defined as in Eq. (12). Finally, if the process exceeds the decision interval, which is contained between the positive and negative sum, then it is assumed, that the process is out of control:

$$\mathbf{C}\_{l}^{+} = \max\left(0, \overline{X}\_{l} \mathbf{-} (\mu\_{0} + \mathbf{K}) + \mathbf{C}\_{l-1}^{+}\right) \tag{11}$$

$$C\_I = \max\left(0, (\mu\_0 + K) \overline{-X}\_i + C\_{i-1}\right) \tag{12}$$

where k is a target value, C<sup>þ</sup> <sup>0</sup> <sup>¼</sup> <sup>C</sup><sup>−</sup> <sup>0</sup> ¼ 0, Ci is cumulative sum for sample i.

#### 4. Results

#### 4.1. BTC/USD

The main goal of this research was to compare the variability of two exchange rates: bitcoin to US Dollar (BTC/USD) and Euro to US Dollar (EUR/USD). The average exchange rates for the quarterly data between 2010 and 2016 will be taken into consideration. The final result of the appliance of the xbar-s chart for BTC/USD is presented in Figure 2.

It is visible that the process is out of control and the layout of the chart is strongly affected by the significant volatility exchange rate increase in 2013. In the period before, it was a long run of very low prices. After the peak in 2013, the level of the exchange rate has never reached a comparable value. Taking the above into consideration, the whole investigated period should be divided into consistent and disjoint periods, i.e. the covering time before the positive price shock, namely the year 2013 and the period after 2013. An attempt to analyse the whole period can be misleading due to the faultily estimated control lines understood as a process average and standard deviation.

Figure 2. xbar-s chart BTC/USD 2010-2016.

<sup>R</sup> <sup>¼</sup> <sup>∑</sup><sup>k</sup>

206 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

where Ri is a range in sample i, k is the number of samples, m is a multiplier chosen to establish control limits, usually set to 3 and d<sup>3</sup> is a constant and σ^ is an estimated variance of a process. CUSUM control charts plot the cumulative sum of deviation from the assumed target value

> Ci <sup>¼</sup> ∑ i

j¼1

where Ci is a cumulative sum, xj is mean of a process in sample j, average and μ<sup>0</sup> is target

If the process is under control, then a cumulative sum (Ci) follows a random walk process with mean equals 0 [35]. It is assumed that the process is out of control, if the average values drift from the target value. If the values move in the positive direction, then the upper cumulative

If, however, they move into the negative values, then the lower cumulative sum is defined as in Eq. (12). Finally, if the process exceeds the decision interval, which is contained between the

The main goal of this research was to compare the variability of two exchange rates: bitcoin to US Dollar (BTC/USD) and Euro to US Dollar (EUR/USD). The average exchange rates for the quarterly data between 2010 and 2016 will be taken into consideration. The final result of the

It is visible that the process is out of control and the layout of the chart is strongly affected by the significant volatility exchange rate increase in 2013. In the period before, it was a long run

0, Xi−ðμ<sup>0</sup> þ KÞ þ C<sup>þ</sup>

<sup>0</sup>,ðμ<sup>0</sup> <sup>þ</sup> <sup>K</sup>Þ−Xi <sup>þ</sup> <sup>C</sup><sup>−</sup>

<sup>0</sup> ¼ 0, Ci is cumulative sum for sample i.

i−1 

i−1 

positive and negative sum, then it is assumed, that the process is out of control:

C<sup>þ</sup> <sup>I</sup> ¼ max

C− <sup>I</sup> ¼ max

<sup>0</sup> <sup>¼</sup> <sup>C</sup><sup>−</sup>

appliance of the xbar-s chart for BTC/USD is presented in Figure 2.

(see Eq. (10)).

value.

sum is written as in Eq. (11).

where k is a target value, C<sup>þ</sup>

4. Results

4.1. BTC/USD

<sup>i</sup>¼<sup>1</sup>Ri

<sup>k</sup> (7)

ðxj−μ0Þ (10)

(11)

(12)

LCL ¼ R−md3σ^ (8)

LCL ¼ R þ md3σ^ (9)

The first studied period was the initial phase of the development of bitcoin. Within that time, the exchange rate has changed significantly, starting from 0.08 USD for 1 BTC, reaching 13.51 USD for 1 BTC at the end of 2012. The xbar-s chart (Figure 3) has generated the CL at the level of 0.62. The layout of the chart suggests that the process being out of control in the period reaching April 2011, when the average volatility was significantly under the LCL equalling to 0.31.

Figure 3. xbar-s chart BTC/USD 2010-2012.

This is justified, as within that time, the price was significantly lower than in other periods. Small changes in prices had almost no effect on volatility. The numbers for May and June have breached the UCL, which equals to 0.93. This peak was connected with strong price jumps, up to 18.50 USD for BTC. The other periods were relatively stable.

A permanent breach of control limits on both sides of CL may be observed, but the deviations are not essential. It is worth to mention that between the years 2010–2012, the graph may imply a pattern. The observed standard deviations tend to move in the same direction almost every 4 months. The extension to the bitcoin volatility analysis gives the MR chart (Figure 4) and CUSUM chart for the standard deviation (Figure 5). At the beginning of the investigated process, the exchange rate volatility was dropping constantly. Commencing in April 2011 it is starting to grow, in par with the price increase. After a strong peak in price, which has also a solid and positive effect on price volatility, the bitcoin faced a volatility decrease at the end of 2011, after which the process started to normalize. This phase lasted until the next shock at the end of 2012, where the volatility has started to grow again.

Figure 4. MR chart BTC/USD 2010-2012.

Figure 5. CUSUM BTC/USD 2010-2012.

In 2013, the BTC/USD sudden price jump was observed, which has affected the average price volatility strongly. Throughout the year 2013, the exchange rate was developing steadily (see Figure 6). The jump took place between October and November. In this period, exchange rate has risen from 155 to 870. The average price volatility for the year 2013 equals to 51.87. If only the first 10 months of the year were considered, then the CL would be at the level of 12.65.

Figure 6. xbar-s chart BTC/USD 2013.

a pattern. The observed standard deviations tend to move in the same direction almost every 4 months. The extension to the bitcoin volatility analysis gives the MR chart (Figure 4) and CUSUM chart for the standard deviation (Figure 5). At the beginning of the investigated process, the exchange rate volatility was dropping constantly. Commencing in April 2011 it is starting to grow, in par with the price increase. After a strong peak in price, which has also a solid and positive effect on price volatility, the bitcoin faced a volatility decrease at the end of 2011, after which the process started to normalize. This phase lasted until the next shock at the

end of 2012, where the volatility has started to grow again.

208 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Figure 4. MR chart BTC/USD 2010-2012.

Figure 5. CUSUM BTC/USD 2010-2012.

Moreover, the MR chart (Figure 7) shows that within the period between January and October 2013 ranges have fluctuated at the zero line until the exchange rate jumps, which caused an increase of almost 150, thereby, the whole process is said to be significantly out of balance.

Figure 7. MR chart BTC/USD 2013.

Figure 8. xbar-s chart BTC/USD 2014-2016.

Bitcoin development during the period (2014–2016) was more stable compared to the previous years. The value for the central line in s-chart (Figure 8) has declined to 28.4 compared with the year 2013. At the beginning and at the end of the investigated period the price fluctuations were observed. In January 2014, the exchange rate has reached the level of 649, which was the continuation of the price increase from the year 2013 and in 2016, when the price reached again the limit of 687 US Dollars for a Bitcoin.

The s-chart has produced warning signals concerning a high volatility between February and May 2014 and for June 2016 by breaching the UCL (45.25), as well as signals for low volatility between March and September 2015 by breaching the LCL (11.53). This downturn shift in price was also reflected by the CUSUM chart (Figure 9).

These have caused a shift in a process, which was followed by the constant movement with the decreasing tendency of deviations from the process mean until the end of the considered period.

Figure 9. CUSUM chart BTC/USD 2014-2016.

#### 4.2. EUR/USD

The USD/EUR exchange rate volatility seems to be more stable within whole investigated period (see Figure 10) compared to the results for BTC/USD (Figure 2). The CL generated by the s-chart was at the level of 0.012. The volatility during the years 2010 and 2011 passes the UCL. Global financial crisis has impacted EUR/USD exchange rates strongly. After the crisis hit Greece in 2009 it has moved on and inadvertently affected other European countries, e.g. Spain and Italy. This was the most significant and long-lasting process disruption signalled by the control chart. The other breaches, which appeared in 2014 and at the beginning of 2015, are very close to the LCL (= 0.0052) and UCL (= 0.02). The deviation from CL seems to be not substantial, especially considering the process standard deviation at the level of 0.002.

This development was also reflected by the CUSUM chart (Figure 11), where mainly the above average values for the standard deviation were signalled in the year 2010 and the beginning of the year 2011. The rest of the process despite the visible downturn trend remained between the UCL (0.029) and LCL (−0.029).

The central line, which reflects the average standard deviation for the entire process equals 0.1272. If the time after 2013 was considered, then the average standard deviation would be equal to 0.1040, hence both results were at a comparable level.

The EUR/USD exchange rate volatility starting from June 2010 was in the downward trend, which ended in December 2014. The s-chart has produced warning signals for points beyond the control limits, which were set at a level of 0.02027 for the UCL and 0.00516 for the LCL, respectively. The first signals were produced for the period between August 2010 and October 2011. Strong fluctuations in this time were caused by weak economic data mainly from the United States. It is worth to mention that this was the period short after the economic crisis, when most of the world economies were unstable.

Figure 10. xbar-s chart EUR/USD 2010-2016.

were observed. In January 2014, the exchange rate has reached the level of 649, which was the continuation of the price increase from the year 2013 and in 2016, when the price reached again

The s-chart has produced warning signals concerning a high volatility between February and May 2014 and for June 2016 by breaching the UCL (45.25), as well as signals for low volatility between March and September 2015 by breaching the LCL (11.53). This downturn shift in price

These have caused a shift in a process, which was followed by the constant movement with the decreasing tendency of deviations from the process mean until the end of the considered period.

The USD/EUR exchange rate volatility seems to be more stable within whole investigated period (see Figure 10) compared to the results for BTC/USD (Figure 2). The CL generated by the s-chart was at the level of 0.012. The volatility during the years 2010 and 2011 passes the UCL. Global financial crisis has impacted EUR/USD exchange rates strongly. After the crisis hit Greece in 2009 it has moved on and inadvertently affected other European countries, e.g. Spain and Italy. This was the most significant and long-lasting process disruption signalled by the control chart. The other breaches, which appeared in 2014 and at the beginning of 2015, are very close to the LCL (= 0.0052) and UCL (= 0.02). The deviation from CL seems to be not

substantial, especially considering the process standard deviation at the level of 0.002.

equal to 0.1040, hence both results were at a comparable level.

This development was also reflected by the CUSUM chart (Figure 11), where mainly the above average values for the standard deviation were signalled in the year 2010 and the beginning of the year 2011. The rest of the process despite the visible downturn trend remained between the

The central line, which reflects the average standard deviation for the entire process equals 0.1272. If the time after 2013 was considered, then the average standard deviation would be

The EUR/USD exchange rate volatility starting from June 2010 was in the downward trend, which ended in December 2014. The s-chart has produced warning signals for points beyond

the limit of 687 US Dollars for a Bitcoin.

4.2. EUR/USD

Figure 9. CUSUM chart BTC/USD 2014-2016.

UCL (0.029) and LCL (−0.029).

was also reflected by the CUSUM chart (Figure 9).

210 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Figure 11. CUSUM chart EUR/USD 2010-2016.

## 5. Conclusion and further work

Bitcoin, a virtual currency, seems to be a promising alternative to a traditional means of payment. According to a survey published by the IMF, bitcoin has many advantages like low transition costs. It offers the possibility to make transactions with countries with weak financial infrastructure and might contribute to transferring developed technologies and solutions to undeveloped countries. At this point, it is worth mentioning that the idea on which the bitcoin is based has a huge potential in many areas, such as banking, accounting, data gathering and transfer, etc. At the same time, it is not without flaws. Reports and surveys concerning bitcoin mention money laundering issues, low recognisability and lack of stability.

In this chapter, an attempt was made to compare the exchange rate volatility between EUR/ USD and BTC/USD. The analysis has shown that the EUR/USD exchange rate volatility is much more stable compared to the BTC/USD exchange rate. In the entire investigated period, the average exchange rate volatility was at the level of 0.013. The first period after the world crisis, when the international economy was unstable and USA has published economic reports below market expectations, the exchange rate volatility has recorded an increase and charts have produced warning signals for the process being out-of-control. In the remaining period, the exchange rate volatility development did not behave in an unpredictable pattern. The downward trend is visible, but no significant shocks were observed. The lack of sudden fluctuations characterises mature economies.

BTC/USD exchange rate volatility is developing in a completely different way. Figure 2 shows three establishing phases of this virtual currency; the first one before 2013, when the price level and the overall recognition were nearly zero. The average volatility was equal to 0.6. This value was affected by the price increase from June 2011. In the year 2013, another strong price jump was visible, when the exchange rate has risen from 155 to 870. The third period under consideration was also susceptible to price shocks, namely at the beginning of 2014 and in June of 2016. This unexpected price increase has strongly affected the average exchange rate volatility of BTC/USD. What is positive, the declining tendency in volatility can observe.

In 2013, the average process volatility including outliers caused by price jumps was equal to 51.8, between 2014 and 2016 it has decreased to 28.4. The xbar-s chart and MR-chart for BTC/USD have showed that except for the above-mentioned price fluctuations the process was most of the time under control. It is worth noting at this point that the range between the upper and lower control limits for the bitcoin is broad, which was caused by the extreme price movements. Because of this lack of stability, it is difficult to model bitcoin behaviour. As such, an attempt to forecast its future behaviour based on its past values would be impossible.

The lack of BTC/USD exchange rate predictability and its excessive volatility causes that at least at this stage of development bitcoin cannot threaten the traditional and regarded as stable, currencies such as USD or EUR. It is also associated with a small recognisability and is still limited to a number of places where it can be exchanged or traded. The bitcoin's founder Satoshi Nakamoto said that he is not sure if the bitcoin in this form will survive, but he is convinced that virtual currencies will exist in the future, in this form or in a different one, due to the increasing loss of confidence and trust among the business partners [36].

## Author details

Beata Szetela

Address all correspondence to: b.rebisz@gmx.de

Ignacy Łukasiewicz Rzeszow University of Technology, Rzeszów, Poland

## References

much more stable compared to the BTC/USD exchange rate. In the entire investigated period, the average exchange rate volatility was at the level of 0.013. The first period after the world crisis, when the international economy was unstable and USA has published economic reports below market expectations, the exchange rate volatility has recorded an increase and charts have produced warning signals for the process being out-of-control. In the remaining period, the exchange rate volatility development did not behave in an unpredictable pattern. The downward trend is visible, but no significant shocks were observed. The lack of sudden

BTC/USD exchange rate volatility is developing in a completely different way. Figure 2 shows three establishing phases of this virtual currency; the first one before 2013, when the price level and the overall recognition were nearly zero. The average volatility was equal to 0.6. This value was affected by the price increase from June 2011. In the year 2013, another strong price jump was visible, when the exchange rate has risen from 155 to 870. The third period under consideration was also susceptible to price shocks, namely at the beginning of 2014 and in June of 2016. This unexpected price increase has strongly affected the average exchange rate volatility of BTC/USD. What is positive, the declining

In 2013, the average process volatility including outliers caused by price jumps was equal to 51.8, between 2014 and 2016 it has decreased to 28.4. The xbar-s chart and MR-chart for BTC/USD have showed that except for the above-mentioned price fluctuations the process was most of the time under control. It is worth noting at this point that the range between the upper and lower control limits for the bitcoin is broad, which was caused by the extreme price movements. Because of this lack of stability, it is difficult to model bitcoin behaviour. As such, an attempt to forecast its future behaviour based on its past values

The lack of BTC/USD exchange rate predictability and its excessive volatility causes that at least at this stage of development bitcoin cannot threaten the traditional and regarded as stable, currencies such as USD or EUR. It is also associated with a small recognisability and is still limited to a number of places where it can be exchanged or traded. The bitcoin's founder Satoshi Nakamoto said that he is not sure if the bitcoin in this form will survive, but he is convinced that virtual currencies will exist in the future, in this form or in a different one, due to the increasing loss of confidence and trust among the business

fluctuations characterises mature economies.

212 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

tendency in volatility can observe.

would be impossible.

partners [36].

Beata Szetela

Author details

Address all correspondence to: b.rebisz@gmx.de

Ignacy Łukasiewicz Rzeszow University of Technology, Rzeszów, Poland


[29] Gronwald, M. The Economics of Bitcoins—Market Characteristics and Price Jumps. CESifo Area Conference on Macro, Money and International Finance. Munich. 20– 21.02.2015

[16] Bryans, D. Bitcoin and money laundering: Mining for an effective solution. Indiana Law Journal, 2014, 89(1). [Updated: 2016.08.17]. Available from: http://www.repository.law.

[17] Plassaras, N. A. Regulating digital currencies: Bringing bitcoin within the reach of IMF.

[18] ECB. Virtual Currency Schemes. [Updated: 2016.08.17]. 2012. Available from: https://www.

[19] ECB. Virtual Currency Schemes—A Further Analysis. Frankfurt am Main: European Central Bank. [Updated: 2016.08.17]. 2015. Available from: https://www.ecb.europa.eu/

[20] Draghi, M. A letter from the President of the ECB to Mr Buonanno, the Honourable Member of the European Parliament, regarding the relevance of virtual currency schemes (VCS) for the ECB's tasks and their potential impact on policies in these specific areas. [Updated: 2016.08.17]. 2015. Available from: http://www.ecb.europa.eu/pub/pdf/other/

[21] Murphy, E. V., Murphy, M., Seitzinger, M. Bitcoin: Questions, Answers and Analysis of Legal Issues. [Updated: 2016.08.17]. 2015. Available from: https://www.fas.org/sgp/crs/

[22] Congress, T. L. Regulation of Bitcoin in Selected Jurisdictions. Global Legal Research Directorate Staff. The Law Library of Congress. [Updated: 2016.08.17]. 2014. Available

[23] Gans, J. S., Halaburda, H. Some Economics of Private Digital Currency. Bank of Canada Working Paper(38). [Updated: 2016.08.17]. 2013. Available from: http://www.

[24] Bank of Canada. Decentralized E-Money (Bitcoin) [Internet]. 2014. Available from: http:// www.bankofcanada.ca/wp-content/uploads/2014/04/Decentralize-E-Money.pdf

[25] Chiu, J., Wong, T.N. On the essentiality of E-money. Staff Working Paper(43). 2015. Available from: http://www.bankofcanada.ca/wp-content/uploads/2015/11/wp2015-43.

[26] Dyhrberg, A.H. Hedging capabilities of bitcoin. Is it the virtual gold? Finance Research

[27] Dyhrberg, A. H. Bitcoin, gold and the dollar—A GARCH volatility analysis. Finance

[28] Cheah, E.T., Fry, J. Speculative bubbles in bitcoin markets? An empirical investigation into the fundamental value of Bitcoin. Economics Letters, 2015, 130, pp. 32–36.

from: http://www.loc.gov/law/help/bitcoin-survey/regulation-of-bitcoin.pdf

bankofcanada.ca/wp-content/uploads/2013/11/wp2013-38.pdf

indiana.edu/ilj/vol89/iss1/13

214 Quality Control and Assurance - An Ancient Greek Term Re-Mastered

150421letter\_buonanno\_3.en.pdf

misc/R43339.pdf

[Accessed: 2016.08.17]

pdf [Accessed: 2016.08.17]

Letters, 2016, Vol. 16. pp. 139–144. .

Research Letters, 2016, Vol. 16. pp. 85–92.

Chicago Journal of International Law, 2013, 14, 377–408.

pub/pdf/other/virtualcurrencyschemesen.pdf abgerufen

ecb.europa.eu/pub/pdf/other/virtualcurrencyschemes201210en.pdf


## *Edited by Leo D. Kounis*

Quality control and assurance cover a diverse area of modern life and play, undeniably, an important role. This book brings together a collection of international papers that showcase examples of current research and practice in industry and the medical profession. It is hoped that engineers, researchers and scientists will be assisted in their continuous quest for excelling in qualitative aspects. The Ancient Greek word arete means excellence or virtue and defines the highest qualitative state: a man's effectiveness and skill in goodness (optimum potentiae). Indeed, Ancient Greeks believed that without quality control, specifications are useless and may result to illegitimacy, which in turn may become a threat to society itself.

Quality Control and Assurance - An Ancient Greek Term Re-Mastered

Quality Control

and Assurance

An Ancient Greek Term Re-Mastered

*Edited by Leo D. Kounis*

Photo by Milkas / iStock