**2. Materials and methods**

Systematic reviews and meta-analyzes are increasingly crucial in the area of health care. In order to optimize meta-analysis reports, an international group developed in 1996 a guide called the QUOROM Statement (Quality Of Reporting Of Meta-analyses), which focused on information from Meta-analysis of a randomized controlled trial [35]. One reason for changing the name from QUOROM to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) was the desire to encompass both systematic reviews and meta analyses.

To improve the literature revision processes Quorum evolved until arriving at the PRISMA1 declaration that consists of a checklist of 27 elements and a diagram of four-phase flow that is applied in this study.<sup>2</sup> The PRISMA statement helps authors improving the reports of systematic reviews and meta-analyzes. It is used as a basis to inform systematic reviews of other types of research.

**35**

**Table 1.**

*A Systematic Review of Usability and Accessibility in Tele-Rehabilitation Systems*

forms for patients who need to perform a tele-rehabilitation?

and exclusion criteria when selecting the articles, such as:

ings of a conference, we chose the complete version, only.

AND recordAbstract:(tele-rehabilitation) AND keywords.author.keyword:(usability heuristic

In order to study the current state of Web platforms for tele-rehabilitation of patients with motor disability, we undertook a systematic review of the literature following the procedures specified in PRISMA. Based on that study, we formulate

1.Which studies currently exist on usability in the development of Web plat-

2.Which studies currently exist related to Web accessibility in the development

The bibliographic search of usability and accessibility in the context of telerehabilitation systems was conducted using the ACM, Google Scholar, IEEE Xplore, and Scopus databases. **Table 1** shows the search chains used for each digital library, in order to obtain as many relevant studies as possible. Concerning the usability and accessibility, we limited our research to publications from 1990

There are many studies related to Web accessibility. Therefore, to ensure that the search is manageable and focused on tele-rehabilitation, we defined some inclusion

• If the studies had been published in more than one journal or in the proceed-

• We excluded non-scientific conferences and journals and papers that are out of

acmdlTitle:(web accessibility) AND recordAbstract:(telerehabilitation) AND keywords.author.keyword:(web

(("Document Title": accessibility OR "Document Title":"web" OR Document Title":"web accessibility") OR

("Publication Title": web accessibility OR "Publication Title":telerehabilitation)) AND ("Abstract": tele-rehabilitation) AND ("Author Keywords":web accessibility OR "Author Keywords":telerehabilitation)

intitle: accessibility OR "web" OR "tele

(SRCTITLE (accessibility OR "web") OR TITLE ( accessibility OR "web") AND ABS (tele-rehabilitation\*) AND KEY (accessibility OR "web") AND

accessibility)

rehabilitation"

(PUBYEAR > 1989)

**Usability search Accessibility search**

of inclusive platforms for patients who require a tele-rehabilitation?

*DOI: http://dx.doi.org/10.5772/intechopen.85869*

the following research questions:

the scope of the review.

evaluation)

*Search strings used for the literature review.*

ACM acmdlTitle:(usability heuristic evaluation)

(("Document Title":usability OR "Document Title":"heuristic evaluation" OR "Document Title":"user experience") OR ("Publication Title":usability OR "Publication Title":"heuristic evaluation" OR "Publication Title":"user experience")) AND ( "Abstract":tele-rehabilitation) AND ( "Author Keywords":usability OR "Author Keywords":"heuristic evaluation" OR "Author

Keywords":"user experience")

Scopus (SRCTITLE (usability OR "heuristic evaluation"

intitle:usability OR "heuristic evaluation" OR "User experience" "tele rehabilitation"

OR "user experience") OR TITLE (usability OR "heuristic evaluation" OR "user experience")) AND ABS (tele-rehabilitation\*) AND KEY (usability OR "heuristic evaluation" OR "user experience") AND (PUBYEAR > 1989)

to 2019.

**Digital Library**

IEEE Xplore

Google Scholar

<sup>1</sup> http://www.prisma-statement.org/

<sup>2</sup> http://prisma-statement.org/prismastatement/flowdiagram.aspx

In order to study the current state of Web platforms for tele-rehabilitation of patients with motor disability, we undertook a systematic review of the literature following the procedures specified in PRISMA. Based on that study, we formulate the following research questions:


The bibliographic search of usability and accessibility in the context of telerehabilitation systems was conducted using the ACM, Google Scholar, IEEE Xplore, and Scopus databases. **Table 1** shows the search chains used for each digital library, in order to obtain as many relevant studies as possible. Concerning the usability and accessibility, we limited our research to publications from 1990 to 2019.

There are many studies related to Web accessibility. Therefore, to ensure that the search is manageable and focused on tele-rehabilitation, we defined some inclusion and exclusion criteria when selecting the articles, such as:

• If the studies had been published in more than one journal or in the proceedings of a conference, we chose the complete version, only.


• We excluded non-scientific conferences and journals and papers that are out of the scope of the review.

#### **Table 1.**

*Assistive and Rehabilitation Engineering*

the user.

29 compliance criteria.

2012, the ISO/IEC 40500. Complementary to these guidelines are the W3C User Agent Accessibility guidelines [29] (UAAG) and Authoring tool Accessibility guidelines [30] (ATAG), which addresses the current technological capabilities to modify the presentation based on the device capabilities and the preferences of

The World Wide Web Consortium (W3C) provides international standards to make the Web as accessible as possible. It comprises the Web 2.0 Content Accessibility Guidelines (WCAG 2.0) [31], also known as the ISO 40500 [32],

The current version of the accessibility guidelines is "Web Content Accessibility Guidelines 2.1" (WCAG 2.1) [23]. WCAG 2.1 consists of 4 principles, 13 guidelines

Principle 1—perceptibility: refers to the good practices regarding the presentation of information and user interface components. It consists of 4 guidelines and

Principle 2—operability: the components of the user interface and navigation

Principle 4—robustness: the content must be robust enough to rely on the interpretation of a wide variety of user agents, including assistive technologies. It

Systematic reviews and meta-analyzes are increasingly crucial in the area of health care. In order to optimize meta-analysis reports, an international group developed in 1996 a guide called the QUOROM Statement (Quality Of Reporting Of Meta-analyses), which focused on information from Meta-analysis of a randomized controlled trial [35]. One reason for changing the name from QUOROM to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) was the desire to encompass both systematic reviews and meta

To improve the literature revision processes Quorum evolved until arriving at

improving the reports of systematic reviews and meta-analyzes. It is used as a basis

declaration that consists of a checklist of 27 elements and a diagram of

The PRISMA statement helps authors

Principle 3—comprehensibility: the information and user interface management

Usability and accessibility can be combined to achieve the development of more accessible, efficient, equitable and universal tele-rehabilitation systems. This chapter presents a systematic literature review of summative and formative usability studies as well as accessibility studies in the context of tele-rehabilitation systems. The remaining of the manuscript is composed of four sections. Section 2 presents the method used to proceed with the systematic review. Section 3 is a description of the most relevant papers in usability applied to tele-rehabilitation. Section 4 describes the results regarding the accessibility. And Section 5 draws conclusions on

which are adapted to the European Standard called EN 301549 [33].

must be operable. It includes 5 guidelines and 29 compliance criteria.

must be understandable. It has 3 guidelines and 17 compliance criteria.

and 76 compliance criteria. The four principles refer to [34].

includes a guideline and three compliance criteria.

the main findings of this literature review.

four-phase flow that is applied in this study.<sup>2</sup>

to inform systematic reviews of other types of research.

http://prisma-statement.org/prismastatement/flowdiagram.aspx

**2. Materials and methods**

http://www.prisma-statement.org/

**34**

1

2

analyses.

the PRISMA1

*Search strings used for the literature review.*


#### **Table 2.**

*The number of research studies identified.*

After the search was completed, 243 results were obtained for usability, and 76 results were obtained for accessibility (**Table 2**). Then, the title and the abstract of these articles were analyzed to identify their relevance to the topic of usability and accessibility in tele-rehabilitation systems.

We use the Start LaPes3 2.3.4.2 tool to analyze the databases that meet the search criteria and Mendeley Desktop4 1.19.3 to manage all the bibliography, details and appointments. This tool helps in the selection process of the scientific articles in such a way that it allows us to eliminate the duplicate articles, to register/filter the accepted and rejected articles in order to obtain reports that are adequate during the meta-analysis process.

Then, in order to study the current state of usability and accessibility related to tele-rehabilitation systems, we conducted a systematic review of the literature based on the PRISMA procedure [36].

When carrying out the literature review, studies that did not meet the established search criteria or referred to another type of usability/accessibility context were discarded. The phases of the selection process according to the PRISMA-based literature review for both themes, usability and accessibility, are detailed in the next two sections.

### **3. Usability review**

The phases of the selection process for the literature review of papers on usability applied to tele-rehabilitation systems are detailed in **Figure 1**.

The 26 papers that were selected for further analysis are listed in **Table 3**.

Usability studies have a significant impact on the quality of the development of e-health systems. The potential for e-health to improve healthcare is partially dependent on its ease of use [45]. A systematic review presented in 2016 indicates that from 40 telemedicine studies, only 7.5% are focused on evaluating the user experience of the patients [46]. Academics, technical studies and scientific communities or special groups [60] have been conducted to evaluate the usability in e-health systems. [49] propose a methodological framework to perform user experience analysis of e-health technologies for patients' engagements. It considers a set of self-report measurements such as mental workload [61], system usability [20] and negative emotional responses [62], in order to provide descriptive data about the user experience of patients. [45] present a systematic review to identify psychometrically tested questionnaires that measure the usability of e-health tools, and to appraise their generalizability, attribute coverage, and quality. The main usability attributes found during the revisions were: learnability, efficiency, and satisfaction.

**37**

*A Systematic Review of Usability and Accessibility in Tele-Rehabilitation Systems*

Likewise, quality appraisal showed that face/content and construct validity were

*PRISMA 2009 flow diagram chart that shows the selection process of the papers included in the literature* 

In [56], authors study the usability of a rehabilitation system designed to demonstrate, instruct and monitor a therapeutic exercise program. The evaluation of usability was carried out in two stages. The first stage consisted of a formative evaluation to identify initial usability problems, defects or omissions in the system. Secondly, user evaluation was conducted by using VRUSE, a computerized usability questionnaire explicitly designed for the evaluation of virtual reality

Weiss et al. [55] present the development life cycle, including usability testing, of a low-cost remote rehabilitation system based on virtual reality technology. The solution was designed to provide remote rehabilitation of the upper extremities in patients who have had a stroke. The usability study was conducted with a population of eight patients. Usability was measured by the 5-point Short Feedback Questionnaire (SFQ ) [57]. This instrument documents the enjoyment of participants and their perception of success and control while using the system. The perception was evaluated by using the Borg scale [63]. The subjective ratings were

In [54], a methodology for a home-based tele-rehabilitation system was presented. This work was done considering patients who require rehabilitation of the arms after having a stroke. The answers were collected via a series of structured

High levels of usability guarantee the substantial quality-of-life. This leads to the acceptance of tele-rehabilitation systems [58]. Likewise, a usability study, to evaluate patients' rehabilitation-evolution, is presented in [51] conclude that the usability is one of the advantages of the tele-rehabilitation system for both patients

*DOI: http://dx.doi.org/10.5772/intechopen.85869*

the most frequent types of validity assessed.

complemented by game performance scores.

and professionals.

**Figure 1.**

*review for usability.*

applications [59].

<sup>3</sup> http://lapes.dc.ufscar.br/tools/start\_tool

<sup>4</sup> https://www.mendeley.com/download-desktop/

*A Systematic Review of Usability and Accessibility in Tele-Rehabilitation Systems DOI: http://dx.doi.org/10.5772/intechopen.85869*

#### **Figure 1.**

*Assistive and Rehabilitation Engineering*

accessibility in tele-rehabilitation systems.

We use the Start LaPes3

meta-analysis process.

**Table 2.**

**3. Usability review**

criteria and Mendeley Desktop4

*The number of research studies identified.*

based on the PRISMA procedure [36].

After the search was completed, 243 results were obtained for usability, and 76 results were obtained for accessibility (**Table 2**). Then, the title and the abstract of these articles were analyzed to identify their relevance to the topic of usability and

**Digital Library Usability studies Accessibility studies**

IEEE Xplore 1 26 Google Scholar 229 29 Scopus 1 7 ACM 12 14 **Total 243 76**

appointments. This tool helps in the selection process of the scientific articles in such a way that it allows us to eliminate the duplicate articles, to register/filter the accepted and rejected articles in order to obtain reports that are adequate during the

Then, in order to study the current state of usability and accessibility related to tele-rehabilitation systems, we conducted a systematic review of the literature

When carrying out the literature review, studies that did not meet the established search criteria or referred to another type of usability/accessibility context were discarded. The phases of the selection process according to the PRISMA-based literature review for both themes, usability and accessibility, are detailed in the next two sections.

The phases of the selection process for the literature review of papers on usabil-

The 26 papers that were selected for further analysis are listed in **Table 3**. Usability studies have a significant impact on the quality of the development of e-health systems. The potential for e-health to improve healthcare is partially dependent on its ease of use [45]. A systematic review presented in 2016 indicates that from 40 telemedicine studies, only 7.5% are focused on evaluating the user experience of the patients [46]. Academics, technical studies and scientific communities or special groups [60] have been conducted to evaluate the usability in e-health systems. [49] propose a methodological framework to perform user experience analysis of e-health technologies for patients' engagements. It considers a set of self-report measurements such as mental workload [61], system usability [20] and negative emotional responses [62], in order to provide descriptive data about the user experience of patients. [45] present a systematic review to identify psychometrically tested questionnaires that measure the usability of e-health tools, and to appraise their generalizability, attribute coverage, and quality. The main usability attributes found during the revisions were: learnability, efficiency, and satisfaction.

ity applied to tele-rehabilitation systems are detailed in **Figure 1**.

2.3.4.2 tool to analyze the databases that meet the search

1.19.3 to manage all the bibliography, details and

**36**

3

4

http://lapes.dc.ufscar.br/tools/start\_tool

https://www.mendeley.com/download-desktop/

*PRISMA 2009 flow diagram chart that shows the selection process of the papers included in the literature review for usability.*

Likewise, quality appraisal showed that face/content and construct validity were the most frequent types of validity assessed.

High levels of usability guarantee the substantial quality-of-life. This leads to the acceptance of tele-rehabilitation systems [58]. Likewise, a usability study, to evaluate patients' rehabilitation-evolution, is presented in [51] conclude that the usability is one of the advantages of the tele-rehabilitation system for both patients and professionals.

In [56], authors study the usability of a rehabilitation system designed to demonstrate, instruct and monitor a therapeutic exercise program. The evaluation of usability was carried out in two stages. The first stage consisted of a formative evaluation to identify initial usability problems, defects or omissions in the system. Secondly, user evaluation was conducted by using VRUSE, a computerized usability questionnaire explicitly designed for the evaluation of virtual reality applications [59].

Weiss et al. [55] present the development life cycle, including usability testing, of a low-cost remote rehabilitation system based on virtual reality technology. The solution was designed to provide remote rehabilitation of the upper extremities in patients who have had a stroke. The usability study was conducted with a population of eight patients. Usability was measured by the 5-point Short Feedback Questionnaire (SFQ ) [57]. This instrument documents the enjoyment of participants and their perception of success and control while using the system. The perception was evaluated by using the Borg scale [63]. The subjective ratings were complemented by game performance scores.

In [54], a methodology for a home-based tele-rehabilitation system was presented. This work was done considering patients who require rehabilitation of the arms after having a stroke. The answers were collected via a series of structured


#### **Table 3.**

*List of the usability papers selected in this review.*

interviews and seven point Likert-based evaluation questions measuring the level of agreement. Ayoade and Baillie [50] present the results of a randomized controlled study in which the authors investigated the usability and feasibility of a rehabilitation

**39**

*A Systematic Review of Usability and Accessibility in Tele-Rehabilitation Systems*

professionals such as quality of life and the improved knee were studied.

visualization system. This work has employed the principles of user-centered design throughout the development phases of the system. The study was performed with the SUS questionnaire [20]. Additionally, metrics significant for both patients and health

Anton et al. [40] present a successful study of usability of a tele-rehabilitation system called KiReS [53]. The evaluation was made in a real scenario. The patients involved in the study underwent a total hip replacement surgery (THR). The study of the subjective perceptions of the patients when participating in the rehabilitation sessions was conducted through an empirical 5-point Likert scale questionnaire based on 13 statements and described in [48]. The instrument is structured in three sections: (1) the system, (2) the user experience, and (3) the user interface. The questionnaire also explores the prior knowledge that patients have regarding telerehabilitation systems. In general, the criticisms obtained by the patients were on

In [16], the formative usability of the Rutgers Ankle Rehabilitation System (RARS) was studied involving three simultaneous users and a remote monitoring. The authors describe a usability evaluation process where engineers and physicians collaborated collaboratively. The usability study uses both traditional and empirical

of a sensor system called "Arm Usage Coach" (AUC) that provides VibroTactile (VT) feedback if the patient does not move the affected arm above a certain threshold level [42]. The study involved 10 patients who suffered a cardiovascular accident and had a mild to moderate arm impairment. The experiment made use of an AUC device on each wrist. VT feedback was given by the device on the affected arm. The usability was measured by using a semi-structured interview and also the SUS questionnaire. The results of the SUS questionnaire indicate that nine participants responded above 70% and one participant responded below 50%. It means that VT feedback is feasible and the AUC can be used as a tele-rehabilitation device to train

In [64], a web-based portal supporting several clinical activities and the Versatile Integrated System for Tele-rehabilitation video-conferencing system called VISYTER were evaluated through the IBM After-Scenario Questionnaire (ASQ ) [65], the Post-Study System Usability Questionnaire (PSSUQ ) [66], the Telehealth Usability Questionnaire (TUQ ) [47], and two demographic surveys. The use of all these instruments allowed capturing subjective and objective information, resulting

in an effective method to evaluate the usability in tele-rehabilitation systems. The ePHoRt platform, which is a tele-rehabilitation system to support people recovering from a hip replacement surgery, has also been the subject of previous usability studies. The first reported usability study was conducted through the SUS. The overall score was 81 out of 100, which suggests a good usability of the Web application [43]. Next, in [39], the authors assert that the lack of usability may lead to problems of confusion, error, and delay, or even abandonment of the physical therapy. This study was a preliminary analysis based on an empirical heuristic evaluation. Unfortunately, the study was not significant enough since most of the interfaces evaluated do not represent the central aspects of the tele-rehabilitation system. Likewise, the study does not consider the efficiency, effectiveness, and satisfaction of patients. However, the authors suggest making successive evaluations of usability. So, a successive usability study was conducted and presented in [41]. The user experience was measured through two aspects: (1) the tasks completion time, and (2) the SUS questionnaire [20]. The results were correlated to an empirical sociodemographic questionnaire. The overall value of the SUS questionnaire was 76.1 out of 100, which

can be considered an acceptable evaluation of the usability of the platform.

A SUS questionnaire was used in order to study the usability and user acceptance

methods to assess the viability of the design of the interface.

and maintain upper extremity used in daily life tasks.

*DOI: http://dx.doi.org/10.5772/intechopen.85869*

the interfaces of the system [48].

#### *A Systematic Review of Usability and Accessibility in Tele-Rehabilitation Systems DOI: http://dx.doi.org/10.5772/intechopen.85869*

visualization system. This work has employed the principles of user-centered design throughout the development phases of the system. The study was performed with the SUS questionnaire [20]. Additionally, metrics significant for both patients and health professionals such as quality of life and the improved knee were studied.

Anton et al. [40] present a successful study of usability of a tele-rehabilitation system called KiReS [53]. The evaluation was made in a real scenario. The patients involved in the study underwent a total hip replacement surgery (THR). The study of the subjective perceptions of the patients when participating in the rehabilitation sessions was conducted through an empirical 5-point Likert scale questionnaire based on 13 statements and described in [48]. The instrument is structured in three sections: (1) the system, (2) the user experience, and (3) the user interface. The questionnaire also explores the prior knowledge that patients have regarding telerehabilitation systems. In general, the criticisms obtained by the patients were on the interfaces of the system [48].

In [16], the formative usability of the Rutgers Ankle Rehabilitation System (RARS) was studied involving three simultaneous users and a remote monitoring. The authors describe a usability evaluation process where engineers and physicians collaborated collaboratively. The usability study uses both traditional and empirical methods to assess the viability of the design of the interface.

A SUS questionnaire was used in order to study the usability and user acceptance of a sensor system called "Arm Usage Coach" (AUC) that provides VibroTactile (VT) feedback if the patient does not move the affected arm above a certain threshold level [42]. The study involved 10 patients who suffered a cardiovascular accident and had a mild to moderate arm impairment. The experiment made use of an AUC device on each wrist. VT feedback was given by the device on the affected arm. The usability was measured by using a semi-structured interview and also the SUS questionnaire. The results of the SUS questionnaire indicate that nine participants responded above 70% and one participant responded below 50%. It means that VT feedback is feasible and the AUC can be used as a tele-rehabilitation device to train and maintain upper extremity used in daily life tasks.

In [64], a web-based portal supporting several clinical activities and the Versatile Integrated System for Tele-rehabilitation video-conferencing system called VISYTER were evaluated through the IBM After-Scenario Questionnaire (ASQ ) [65], the Post-Study System Usability Questionnaire (PSSUQ ) [66], the Telehealth Usability Questionnaire (TUQ ) [47], and two demographic surveys. The use of all these instruments allowed capturing subjective and objective information, resulting in an effective method to evaluate the usability in tele-rehabilitation systems.

The ePHoRt platform, which is a tele-rehabilitation system to support people recovering from a hip replacement surgery, has also been the subject of previous usability studies. The first reported usability study was conducted through the SUS. The overall score was 81 out of 100, which suggests a good usability of the Web application [43]. Next, in [39], the authors assert that the lack of usability may lead to problems of confusion, error, and delay, or even abandonment of the physical therapy. This study was a preliminary analysis based on an empirical heuristic evaluation. Unfortunately, the study was not significant enough since most of the interfaces evaluated do not represent the central aspects of the tele-rehabilitation system. Likewise, the study does not consider the efficiency, effectiveness, and satisfaction of patients. However, the authors suggest making successive evaluations of usability. So, a successive usability study was conducted and presented in [41]. The user experience was measured through two aspects: (1) the tasks completion time, and (2) the SUS questionnaire [20]. The results were correlated to an empirical sociodemographic questionnaire. The overall value of the SUS questionnaire was 76.1 out of 100, which can be considered an acceptable evaluation of the usability of the platform.

*Assistive and Rehabilitation Engineering*

Platform

Platform

Hip Surgery Patients

Management of Therapies

Rehabilitation Systems

replacement patients

impairment

extremity

usability

and Context

**Table 3.**

virtual environments

synthetic environment systems

*List of the usability papers selected in this review.*

Intervention: The H-CIM Case

An Agile Approach to Improve the Usability of a Physical Telerehabilitation

Implementation and Assessment of an Intelligent Motor Tele-Rehabilitation

A Telerehabilitation System for the Selection, Evaluation and Remote

Interaction with a Tele-Rehabilitation Platform Through a Natural User

USEQ: A Short Questionnaire for Satisfaction Evaluation of Virtual

Validation of a Kinect-based telerehabilitation system with total hip

Evaluating Patient Engagement and User Experience of a Positive Technology

SEQ: Suitability Evaluation Questionnaire for Virtual Rehabilitation Systems. Application in a Virtual Rehabilitation System for Balance Rehabilitation

Assessment and training in home-based telerehabilitation of arm mobility

Usability evaluation of e-motion: a virtual rehabilitation system designed to demonstrate, instruct and monitor a therapeutic exercise program

Short feedback questionnaire (SFQ ) to enhance client-centered participation in

Formative evaluation of a virtual reality telerehabilitation system for the lower

Dimensions of diversity in design of telerehabilitation systems for universal

VRUSE—a computerized diagnostic tool: for usability evaluation of virtual/

Evaluation—Methodology for Telematic Application Systems: Quality for Users

On the Use of Natural User Interfaces in Physical Rehabilitation: a Web-based

Interface: a Case Study of Hip Arthroplasty Patients

Application for Patients with Hip Prosthesis

Analysis and Improvement of the Usability of a Tele-rehabilitation Platform for

**Title Authors Year**

Usability study of a Web-Based Platform for Home Motor Rehabilitation Pérez-Medina et al. [38] 2019

Usability Evaluation of a Vibrotactile Feedback System in Stroke Subjects Held et al. [42] 2017

Towards usable E-health. A Systematic Review of Usability Questionnaires Sousa et al. [45] 2017 A Renewed Framework for the Evaluation of Telemedicine Jansen-Kosterink et al. [46] 2016 Development of the telehealth usability questionnaire (TUQ ) Parmanto et al. [47] 2016

A novel knee rehabilitation system for the home Ayoade et al. [50] 2014 Telerehabilitation web application for health care Eguiluz-Perez et al. [51] 2014

Kires: a Kinect-based telerehabilitation system Anton et al. [53] 2013

Development and validation of tele-health system for stroke rehabilitation Weiss et al. [55] 2012

Pilco et al. [37] 2019

Rybarczyk, et al. [6] 2019

Pilco et al. [39] 2018

Anton et al. [40] 2018

Rybarczyk et al. [41] 2018

Rybarczyk et al. [43] 2017

Gil-Gómez et al. [44] 2017

Anton et al. [48] 2016

Triberti et al. [49] 2015

Gil-Gómez et al. [52] 2013

Perry et al. [54] 2013

Fitzgerald et al. [56] 2008

Kizony et al. [57] 2006

Whitworth et al. [16] 2003

Lathan et al. [58] 2000

Kalawsky et al. [59] 1999

Arnold et al. [60] 1997

**38**

interviews and seven point Likert-based evaluation questions measuring the level of agreement. Ayoade and Baillie [50] present the results of a randomized controlled study in which the authors investigated the usability and feasibility of a rehabilitation

A more recent work conducted on an updated version of ePHoRt synthesizes the results obtained from a study of effectiveness, efficiency, and subjective user satisfaction [38]. Thirty nine participants tested the platform and were asked to evaluate its usability by using the IBM CSUQ. The empirical results based on subjective perception and self-reported feedback show that the application is useful, effective, efficient, easy to use, and its interfaces are acceptable. In addition, the evaluation of the user experience enables us to identify usability aspects that should be implemented, in order to improve the visual interface. The experiment indicates that user guidance is a critical aspect to ensure good usability of the tele-rehabilitation platform. Likewise, the error messages received by users should be as detailed as possible. Finally, in [6] the authors suggest to carry out usability tests with real patients and during the whole completion period of the rehabilitation program, as indicated in [36]. For this purpose, there are usability instruments specifically designed to measure the user experience with e-Health platforms, such as SEQ [52], USEQ [44] and TUQ [47].

Pilco et al. [37] is a very recent usability study that makes use of an agile user centered design process [13, 14] to assess the ePHoRt platform. The evaluation process stated with a heuristic evaluation by using the Nielsen's 10 heuristics principles [67]. In addition, a cognitive workload assessment was performed to complete the usability evaluation. Likewise, users were involved through all the stages of the iterative refinement process. Usability issues were progressively reduced by applying improvements suggested from the iterative assessments. For instance, usability issues originally cataloged as catastrophic were reduced to zero, major usability problems were diminished to 10 (2.75%), and minor usability problems were decreased to 141 (38.74%).
