**2. Conceptual foundation**

different clinical settings [3] that eventually generate quality benefits and minimize medical costs from avoiding unnecessary clinical trials, examinations, and treatments [4]. Therefore, the management and presentation of HISs are vital to accelerate patient care and its continuity across health institutions [5]. The success of system implementation relies upon a high quality of information outputs from HISs required to make timely and accurate clinical decisions by various health practitioners [6]. Besides enabling care continuity, HIS is regarded as the wealthiest source of clinical evidence to support continuous communication among individual clinicians and surgical team works [7]. With the use of HIS, it is not only capable to reduce human errors [8] but also contributes to an increased adherence to clinical guidelines and deterrence of medical errors [9, 10], thereby delivering greater patient safety and medication

In Malaysia, the expenditures of customized HISs are fully supported by the government in the efforts to retain a higher standard of patient care [12, 13]. All new public hospitals should be equipped with HISs designed from multiple vendors hired by the government. Although the investment of IS can improve health service, it will also present more costs in mainte‐ nance, hardware replacements, end‐user trainings, and system upgrades [14, 15]. Increasing medical costs [16] and enormous budget cuts among local hospitals have demanded for a comprehensive evaluation of HIS to investigate the most possible strengths and weaknesses for further improvements. In reality, the effectiveness of HIS adoption among implemented government hospitals had never been assessed since its first kick‐off at Selayang Hospital in 1999. Hospitals with HISs are repeatedly distributing user satisfaction surveys without concentrating on significant success factors and impact on the performance of the health personnel. They conducted these surveys to satisfy the auditing needs but the results were still insufficient in recommending which critical attributes for improving system use and user productivity. As a consequence, the government hospitals were still incapable of choosing the right HIS and vendors and even assessing its performance after implementation [17]. A systematic IS evaluation will not only promote efficient use and medical cost savings but also cope with unresolved issues of clinicians' heavy workloads and shortage of specialists in this

Identifying the needs of health workforce and acknowledging the characteristics of HIS are essential to their productivity that must be emphasized in any evaluation studies [18, 19]. For that reason, recognizing the main attributes of HIS can improve health practitioners' perfor‐ mance from their daily use. Strategies to upgrade an HIS could not precede with an absence of in‐depth knowledge about the most significant HIS characteristics in predicting user productivity. Consequently, there will be wasted expenses on any system upgrades without careful understanding of the potential system impacts or benefits to the user performance,

Unfortunately, there is little evidence on the prior HIS research in measuring the influence of IS attributes toward satisfaction and productivity of medical practitioners [20, 21]. Besides, the previous evaluation works did not completely assess the importance and performance of multiple HIS attributes especially in ranking those attributes with high importance for managerial attention. There are only two current studies attempted to prioritize different HIS

thereby introducing dissatisfaction and risks of system failure [18].

management [11].

34 Advances in Health Management

multi‐racial country [14, 15].

The DeLone and McLean IS success models (DMISMs) are the most outstanding theoretical frameworks adopted by IS researchers since the past two decades for IS evaluation including the health‐care domain [22, 23]. The models embrace system quality, information quality, service quality, actual use, and user satisfaction to predict individual impact, organizational impact, and net benefits [24, 25]. In our empirical study, the traditional DMISM models will be extended to incorporate knowledge quality and effective use in predicting individual per‐ formance based on the perception of medical practitioners as HIS system users.

#### **2.1. Effective use and user performance**

Effective use and user performance are the two outcome constructs measured in our evalua‐ tion study. When actual system use denotes the extent or frequency of HIS usage [26], effective use more refers to the outputs of HIS usage that allows the medical practitioners to complete their clinical tasks easily without any misdiagnosis and inaccurate medication. Because of the mandatory use of HIS, the actual use remains unreliable in assessing IS success [27, 28].

Previous research indicated that user satisfaction had a strong relationship with system quality, information quality, and individual impact [29, 30]. This construct is indeed composed of system quality and individual impact measures [31] that finally disclosed a little explanatory power [32]. Consequently, user satisfaction is omitted as the outcome construct in the study.

On the other hand, individual impact is the outcome generated by IS workers from their applied IT knowledge, skills, and experiences [33]. Likewise, user performance in this study refers to the level to which the practitioners gain benefits from the effective use of HIS by considering patient care and safety, work productivity, and performance score.

#### **2.2. Predictors of health information system evaluation**

System quality is the attributes or characteristics of HIS including functionality, features, inter‐ face design, and its performance to facilitate ease of clinical task completion [34]. With regard to past empirical studies on the most important predictors of HIS quality [19, 22, 23, 35], we will limit the scope of measuring this predictor with the four measures namely adequate IT infra‐ structure, system interoperability, perceived security concerns, and system compatibility.

In the conventional DMISMs, information quality describes the usable, meaningful, and understanding the content and format of IS outputs [24, 36]. Clinicians can deliver the right care depending on the quality of information produced from HIS [37]. For that reason, successful adoption of HIS is determined from the quality of records it produced [7]. The researchers will more specify the generic term of information quality with records quality based on timely access, consistency, standardized, accuracy, duplication prevention, and completeness of patient notes, reports, prescriptions, images, laboratory test results, and discharge summaries.

In general, service quality is about the type of IS support delivered by the responsible IS providers or personnel [38]. We will extend service quality construct with quick assistance, problem‐solving capability, follow‐up service, and adequate training in the study.

The advancement of interoperable HISs from time to time will not only create, store, and man‐ age data and information but also knowledge [12, 35]. The aim of HIS adoption in most hospitals is to acquire, classify, store, access, and simplify the use of knowledge from a HIS repository of patient health information for supporting clinical decision‐making, actions, and problem solving [39, 40]. Besides, HISs can be utilized to promote knowledge management activities in a health organization through medical research and education [41]. In essence, medical knowledge is clas‐ sified into two types such as tacit and explicit. Tacit knowledge is gathered through professional practices and experiences of medical practitioners while explicit knowledge is generally embed‐ ded and presented into the forms of electronic health records (EHRs), electronic medical records (EMRs), clinicians' workflows, clinical guidelines, and protocols [42, 43]. HIS also integrates clini‐ cal decision support system (CDSS) and computerized provider order entry (CPOE) as the knowl‐ edge tools to hold medical knowledge [39, 42–44]. It should be noted that the wide adoption of HIS worldwide is not only due to EHRs but also its integration with CDSS and CPOE to raise higher quality of patient care [45]. Hence, the quality of knowledge must be included in any HIS evaluation [12, 41]. As a new measuring predictor in this study, knowledge quality is defined as the level to which the medical practitioners believe that using HIS will increase their medical knowledge and competencies [41] and then practice it to deliver the best patient care.

### **3. Empirical example**

Our study would bridge the knowledge gap with current empirical proof in the local health system to determine the importance and performance of several effectiveness factors for immediate mana‐ gerial actions with regard to the effective use of HISs and medical practitioners' performance as the measuring outcomes. The research design would employ a quantitative method with the distribu‐ tion of survey questionnaire to the four groups of health personnel in the three different government hospitals with multiple HISs. By utilizing importance‐performance map analysis (IPMA) feature in partial least squares structural equation modeling (PLS‐SEM), the expected outcomes could establish the most critical quality attributes for effective use and user performance improvements.

An ethic approval was obtained from the Medical Research and Ethics Committee Malaysia as the study engaged the human subject responses from varying clinical professionals. Subsequently, the data were gathered from three hospitals situated in different states with different HIS packages. These hospitals had more than 1000 health personnel with more than 500 beds for patients. Specifically, Kedah Hospital used iSOFT system, Pahang Hospital used F1S1C1EN® system, and Johor Hospital used Cerner system. Connected via a centralized and secured 1Gov\*Net network, all HISs are integrated with various clinical modules including patient management, laboratory, radiology, pharmacy, picture archiving and communication, nursing, and operating theater management. The implemented systems are in the current phase of operation and maintenance while the contract is renewed for every 3 years. The gov‐ ernment did not standardize the use of single HIS package across their administered hospitals in order to avoid monopoly by a sole vendor that will render a negative image to the public.

limit the scope of measuring this predictor with the four measures namely adequate IT infra‐ structure, system interoperability, perceived security concerns, and system compatibility.

In the conventional DMISMs, information quality describes the usable, meaningful, and understanding the content and format of IS outputs [24, 36]. Clinicians can deliver the right care depending on the quality of information produced from HIS [37]. For that reason, successful adoption of HIS is determined from the quality of records it produced [7]. The researchers will more specify the generic term of information quality with records quality based on timely access, consistency, standardized, accuracy, duplication prevention, and completeness of patient notes, reports, prescriptions, images, laboratory test results, and

In general, service quality is about the type of IS support delivered by the responsible IS providers or personnel [38]. We will extend service quality construct with quick assistance,

The advancement of interoperable HISs from time to time will not only create, store, and man‐ age data and information but also knowledge [12, 35]. The aim of HIS adoption in most hospitals is to acquire, classify, store, access, and simplify the use of knowledge from a HIS repository of patient health information for supporting clinical decision‐making, actions, and problem solving [39, 40]. Besides, HISs can be utilized to promote knowledge management activities in a health organization through medical research and education [41]. In essence, medical knowledge is clas‐ sified into two types such as tacit and explicit. Tacit knowledge is gathered through professional practices and experiences of medical practitioners while explicit knowledge is generally embed‐ ded and presented into the forms of electronic health records (EHRs), electronic medical records (EMRs), clinicians' workflows, clinical guidelines, and protocols [42, 43]. HIS also integrates clini‐ cal decision support system (CDSS) and computerized provider order entry (CPOE) as the knowl‐ edge tools to hold medical knowledge [39, 42–44]. It should be noted that the wide adoption of HIS worldwide is not only due to EHRs but also its integration with CDSS and CPOE to raise higher quality of patient care [45]. Hence, the quality of knowledge must be included in any HIS evaluation [12, 41]. As a new measuring predictor in this study, knowledge quality is defined as the level to which the medical practitioners believe that using HIS will increase their medical

problem‐solving capability, follow‐up service, and adequate training in the study.

knowledge and competencies [41] and then practice it to deliver the best patient care.

Our study would bridge the knowledge gap with current empirical proof in the local health system to determine the importance and performance of several effectiveness factors for immediate mana‐ gerial actions with regard to the effective use of HISs and medical practitioners' performance as the measuring outcomes. The research design would employ a quantitative method with the distribu‐ tion of survey questionnaire to the four groups of health personnel in the three different government hospitals with multiple HISs. By utilizing importance‐performance map analysis (IPMA) feature in partial least squares structural equation modeling (PLS‐SEM), the expected outcomes could establish the most critical quality attributes for effective use and user performance improvements.

discharge summaries.

36 Advances in Health Management

**3. Empirical example**

Adopted from past surveys [36, 38, 41, 46–50] with 19 new item additions anchoring by seven‐point Likert scales from 1 of strongly disagree to 7 of strongly agree, the questionnaire draft was proven valid and reliable after pretesting between key HIS experts and pilot testing among 100 samples of end users using exploratory factor analysis in statistical package for social science (SPSS) software. The field survey data contained 888 samples from specialists, medical officers, and nursing staffs collected by the mean of convenience sampling technique. Overall, 353 participated respondents were from Kedah Hospital, 213 from Pahang Hospital, and 322 respondents from Johor Hospital. Specifically, 71 and 96 were specialists and assis‐ tant medical officers, respectively, 328 were medical officers, and 393 were nurses. More than 70% of respondents were female due to imbalance recruitments of clinical professionals and nurses were majorly female while 64% of total samples aged between 25 and 35 years old. About 53% of assistant medical officers and nurses had Diploma qualifications in medical and nursing, respectively, whereas the remaining 47% medical officers and specialists had Bachelor, Masters, or PhD Degree in medical.

The collected data were subjected to confirmatory factor analysis using SmartPLS software. In this study, system quality characteristics namely adequate IT infrastructure, system interoper‐ ability, perceived security concerns, and system compatibility are identified as the formative measures. The formative model exhibited no collinearity issue for all measuring indicators and passed weight significance at a level of 1%. Then, in the reflective model, all question items satisfy the required outer loadings, composite reliability (CR), and average variance extracted (AVE) scores above the suggested thresholds [51, 52], confirming the convergent validity. However, one attribute of knowledge quality (knowqual\_4) was deleted due to lower factor loading below 0.70.

Discriminant validity was then executed using the Fornell and Larcker [53] criterion, and cross‐loading methods. Every construct average variance extracted is more than 0.50 that sat‐ isfied the required criterion [53, 54] while cross‐loading scores of bolded indicators are higher than its opposing indicators in other constructs [55].

The next assessment was preceded to evaluate the path model. After running a complete boot‐ strapping test with 5000 subsamples and no sign option setting, the PLS results in **Table 1** dem‐ onstrate that the observed path coefficients were statistically significant at either 0.05 or 0.01


**Table 1.** Path coefficients.

level, and had positive effects on the outcomes or target constructs except for service quality and effective use relationships. The outcome of user performance had the largest predictive power explained by quality predictors and effective use. More importantly, knowledge quality as a new predictor became the strongest predictor for user performance at a 1% level of significance. This construct also had large effect size among other predictors that justified a need for measur‐ ing knowledge quality in future system evaluation studies.

The path coefficient scores for each latent construct would be subjected to further assessment in importance‐performance map analysis. IPMA in PLS‐SEM adopts the traditional IPA method in ranking both critical constructs and their measured indicators' importance and performance for managerial intervention [51, 56]. Moreover, PLS‐SEM simplifies the researchers to model both higher‐order constructs and their individual indicators simultaneously for calculating attribute importance scores. It helps to reduce the collinearity issues between the attribute items if using a simple regression analysis [57]. The study results can be valuable in contributing to the practical implications to decision‐makers and administrators by incorporating IPMA. IPMA extends the PLS‐SEM results for path coefficient scores by contrasting the total effects of constructs' importance in measuring target constructs with their average latent scores representing their performance.

In a graphical representation, IPMA contrasts the (unstandardized) total effects on the horizontal axis with the latent construct scores, rating on a scale of 0–100, on the vertical axis. The estimated results will be emphasized on the bottom of IPMA diagram [58]. The key objective of this analysis is to improve the performance of constructs with greater importance (strong total effect) but lower performance (small construct score) in predicting a single or more target constructs [51, 55]. Hence, the subsequent analysis would apply IPMA to highlight which latent constructs and their manifest attributes necessary for remedial attentions by both decision‐makers and hospital administrators.

The IPMA diagram in **Figure 1** exhibits system quality has the strongest total effect over the outcome construct. Consequently, knowledge quality, records quality, and service quality should be improved to increase the effective use of HISs.

When selecting user performance as a target construct as displayed in **Figure 2**, knowledge quality becomes the highest importance among others. System quality, records quality, service quality, and effective use are deserved for critical managerial attention to enhance the perfor‐ mance of medical practitioners. No underperforming construct below 50% is identified.

Extending Health Information System Evaluation with an Importance‐Performance Map Analysis http://dx.doi.org/10.5772/68122 39

**Figure 1.** IPMA for effective use at construct level.

level, and had positive effects on the outcomes or target constructs except for service quality and effective use relationships. The outcome of user performance had the largest predictive power explained by quality predictors and effective use. More importantly, knowledge quality as a new predictor became the strongest predictor for user performance at a 1% level of significance. This construct also had large effect size among other predictors that justified a need for measur‐

**Path coefficients (user performance)** 

*R***‐squared: 0.640**

The path coefficient scores for each latent construct would be subjected to further assessment in importance‐performance map analysis. IPMA in PLS‐SEM adopts the traditional IPA method in ranking both critical constructs and their measured indicators' importance and performance for managerial intervention [51, 56]. Moreover, PLS‐SEM simplifies the researchers to model both higher‐order constructs and their individual indicators simultaneously for calculating attribute importance scores. It helps to reduce the collinearity issues between the attribute items if using a simple regression analysis [57]. The study results can be valuable in contributing to the practical implications to decision‐makers and administrators by incorporating IPMA. IPMA extends the PLS‐SEM results for path coefficient scores by contrasting the total effects of constructs' importance in measuring target constructs with their average latent scores representing their performance.

In a graphical representation, IPMA contrasts the (unstandardized) total effects on the horizontal axis with the latent construct scores, rating on a scale of 0–100, on the vertical axis. The estimated results will be emphasized on the bottom of IPMA diagram [58]. The key objective of this analysis is to improve the performance of constructs with greater importance (strong total effect) but lower performance (small construct score) in predicting a single or more target constructs [51, 55]. Hence, the subsequent analysis would apply IPMA to highlight which latent constructs and their manifest attributes necessary for remedial attentions by both decision‐makers and hospital administrators. The IPMA diagram in **Figure 1** exhibits system quality has the strongest total effect over the outcome construct. Consequently, knowledge quality, records quality, and service quality

When selecting user performance as a target construct as displayed in **Figure 2**, knowledge quality becomes the highest importance among others. System quality, records quality, service quality, and effective use are deserved for critical managerial attention to enhance the perfor‐ mance of medical practitioners. No underperforming construct below 50% is identified.

ing knowledge quality in future system evaluation studies.

Significance Level: \*\*\**p* < 0.01, \*\**p* < 0.05.

**Table 1.** Path coefficients.

38 Advances in Health Management

**(effective use)** *R***‐squared: 0.260**

System quality 0.320 (6.025\*\*\*) 0.122 (3.127\*\*\*) Records quality 0.103 (2.115\*\*) 0.137 (3.515\*\*\*) Service quality 0.047 (1.244) 0.139 (4.632\*\*\*) Knowledge quality 0.121 (2.520\*\*) 0.489 (12.464\*\*\*) Effective use ‐ 0.104 (4.170\*\*\*)

should be improved to increase the effective use of HISs.

**Figure 2.** IPMA for user performance at construct level.

As this construct level of analysis does not reveal which specific attributes required for further improvement, a subsequent analysis is continued with the individual measuring items for each latent construct. In **Figure 3**, syscom\_1 (workflows fit) and syscom\_2 (work styles fit) should be maintained for the continued effective use of HISs. By contrast, other quality attri‐ butes that fall into low performance must be stressed for managerial actions. For example, the attribute secc\_4 (secure and save) has an average importance on effective use, while offering room for improving its performance. IT departments can focus on offering hands‐on training to educate HIS users about securing their access when using the systems [18]. In addition, user access control policy should be enforced and applied across the government hospitals with HISs to prevent unauthorized access and misuse of patient health information by non‐responsible doctors. Unfortunately, secc\_1 (unauthorized access) attribute was removed from the analysis due to negative outer weight score in the measurement model assessment as suggested by Ringle and Sarstedt [58].

Next, in **Figure 4**, by retaining knowledge quality for sustaining greater user performance, all effective use, service quality, system quality, and records quality attributes demand for urgent intervention. For instance, indicator effuse\_2 (misdiagnosis prevention) should receive particular attention by promoting HIS adoption across the country so that any misdiagnosis will be averted from timely and full access to comprehensive EHR of every patient. As a result, the importance of effective use increases and then improves user performance outcome. Interestingly, no attribute falls into the bottom zone, signifying that all measuring items for every predictor achieved more than 60% of performance score in the diagram.

**Figure 3.** IPMA for effective use at indicator level.

Extending Health Information System Evaluation with an Importance‐Performance Map Analysis http://dx.doi.org/10.5772/68122 41

**Figure 4.** IPMA for user performance at indicator level.

As this construct level of analysis does not reveal which specific attributes required for further improvement, a subsequent analysis is continued with the individual measuring items for each latent construct. In **Figure 3**, syscom\_1 (workflows fit) and syscom\_2 (work styles fit) should be maintained for the continued effective use of HISs. By contrast, other quality attri‐ butes that fall into low performance must be stressed for managerial actions. For example, the attribute secc\_4 (secure and save) has an average importance on effective use, while offering room for improving its performance. IT departments can focus on offering hands‐on training to educate HIS users about securing their access when using the systems [18]. In addition, user access control policy should be enforced and applied across the government hospitals with HISs to prevent unauthorized access and misuse of patient health information by non‐responsible doctors. Unfortunately, secc\_1 (unauthorized access) attribute was removed from the analysis due to negative outer weight score in the measurement model assessment

Next, in **Figure 4**, by retaining knowledge quality for sustaining greater user performance, all effective use, service quality, system quality, and records quality attributes demand for urgent intervention. For instance, indicator effuse\_2 (misdiagnosis prevention) should receive particular attention by promoting HIS adoption across the country so that any misdiagnosis will be averted from timely and full access to comprehensive EHR of every patient. As a result, the importance of effective use increases and then improves user performance outcome. Interestingly, no attribute falls into the bottom zone, signifying that all measuring items for

every predictor achieved more than 60% of performance score in the diagram.

as suggested by Ringle and Sarstedt [58].

40 Advances in Health Management

**Figure 3.** IPMA for effective use at indicator level.

More specifically, **Table 2** lists the importance and performance scores for every predictor attribute with its discrepancy, calculating by subtracting performance value against importance value [59, 60]. In doing so, performance score in percentage of individual attribute has to be converted into three decimal places before computation. The results confirmed that attribute secc\_3 (robust security control) of the largest discrepancy in effective use warranted for imme‐ diate managerial intervention mainly when the respondents expressed their concerns over lack of security control in HISs. When referring to previous IPMA diagram, this attribute had the lowest total effect (importance) score. Again, a proper security policy must be in place




**Table 2.** Performance and importance scores for individual attribute.

to limit the access level by specific clinical roles. Regular monitoring and reporting of access activities can be further improved with audit trail feature. On‐site training can be empha‐ sized on instructing users by changing passwords frequently with a combination of numbers, alphabets, and symbols as well as securing their accounts through routine check of logging off after using the systems. By contrast, attribute adin\_2 (adequate computers) had the highest discrepancy in user performance outcome, demanding for more computers to use HISs. In coping with a tight budget facing by most hospitals and the increasing rates of doctors, the hospitals may consider to provide grants in purchasing high‐performance desktop and laptop computers at low costs from their contracted system vendors.
