**1.3 App evaluation criteria are not design criteria**

Many organizations and academics have developed criteria for evaluating apps, including the author of this paper [13, 17, 19–21]. However, no matter how comprehensive the criteria, they are both aspirational, incomplete and highly generic. For example, the very popular and highly cited Mobile Apps Rating Scale (MARS) assesses constructs such as engagement, functionality, esthetics, information and subjective quality. However, the MARS has nothing to say about disease-specific functions, integration with the healthcare system, value for healthcare providers, whether the app addresses important barriers to use or even whether the app uses behavioral theories to help with behavior change. The authors of the study even removed the concept of "evidence-based" because it was not assessed during the development of the tool. Interestingly, the authors claim that the MARS can be used as "a checklist for the design and development of new high quality health apps" [19]. The checklist is indeed useful as a general checklist of usability and user experience criteria to keep in mind when building the app but does not provide much guidance on how to design to achieve clinical benefits.

## **1.4 Human factors are necessary, but not sufficient for success**

Most app design frameworks recommend using a human factors design approach [8–12]. These typically include 1) getting some form of input and motivations for use (user stories) from actual users and developing profiles of the different users who might use the system for a shared understanding of users (personas); some models call this the empathize step; 2) collating information from multiple sources to identify important user problems, also called the define phase in some models; 3) developing low and/or medium fidelity mock-ups and working with end users to get feedback on them; also called the ideate phase; 4) develop prototypes of the most promising ideas; called the prototype phase; 5) conducting expert usability, cognitive walkthroughs and end-user testing of the prototypes; called the test phase [22, 23].

Not surprisingly, most mhealth app developers do use human factors methods in their design process [24]. Despite using these methods, usage of mhealth apps continues to be low [4]. This is likely because human factors is only one important ingredient in a complex mix of important ingredients [17].

The International Standards Organization (ISO) has recommended a move away from user-centred design to a more human-centred design approach [25]. The name change is intended to help developers expand their focus from a single user and to start thinking more holistically about who the stakeholders of a product really are. The ISO recommends six components for human-centred design process, including: 1) documenting the needs of users and their relationships to tasks and the contexts in which they occur; 2) obtaining user involvement in all stages of design and software development; 3) obtaining on-going user feedback and conducting on-going evaluation; 4) using an iterative process for product refinement; 5) having multidisciplinary skills and diverse perspectives on the design team; and 6) ensuring the design encompasses the entire user experience (UX).

#### **1.5 What is the role of personas in app development?**

Personas have been an important part of app design for several years [26]. They enable product managers and designers to convey important information about the end-users to the development team. Personas provide product managers a structured approach to convey basic information about the different types of end-users that may use the product. What are the demographics of different users, including age, gender, geography? What are their goals, their spending habits, their needs, their pain points? What is their orientation to technology (avid, phobic or in between)? What is their attitude to the healthcare system (trusting, mistrusting, in between)? What is their relationship to their healthcare provider (want to be told what to do, want to make decisions themselves)? The persona is usually built after conducting research with users and then synthesizing a 'typical profile' of either a single ideal user or a few different types of users [27, 28].

Personas are particularly important in marketing, as they are typically built on well-researched market segments where the desired outcome (purchasing behavior) is well known [29]. Personas are essentially a mechanism to make an abstract segment more concrete for marketers, copy editors and advertising creatives. Personas have been adapted for app development, but they lack one important piece of information that is available to marketers, but not available to app developers: desired user behaviors.

Increasingly, personas are considered as caricatures and not enough for app development [30]. They are being augmented by a more powerful model popularized by Clayton Christenson: The Jobs to be Done [JTBD] framework [31, 32]. The JTBD framework posits that users do not purchase or use products for the product's sake, but rather purchase them to 'get a job done'. They 'hire' products to accomplish a task they need to complete and are looking for an outcome, rather than a feature or a function. The JTBD framework is an outcomes specification approach rather than a functional specification approach [33].

This framework provides better insight for developing functionality, because it immediately goes to the heart of why a user might want to use a product and generates a list of 'jobs to be done'. In the healthcare sector, some of the job to be done includes being able to communicate information to the HCP which the HCP considers to be relevant and easy to interpret, which is not always clear from a end-user persona perspective [34, 35].

The JTBD framework also provides guidance on identifying drivers and barriers, both rational/cognitive and affective, for using a product to get the job done [33]. This assists developers and product managers to consider unspoken barriers which inhibit the use of the product, even when the product is designed to get the 'job done'. It also assists in articulating unidentified drivers that could enhance uptake of the product but were not articulated by the end user during user-centered design sessions [33].

## **1.6 Apps need to follow and adapt to evidence-based guidelines for treatment and outcomes**

One key reason why apps are not used is because the app makes recommendations which conflict with what HCPs communicate or believe [17]. This creates a tension for the patient: "Should I believe the app or should I believe my HCP?" Inevitably, unless the patient can find a HCP who agrees with the app, the patient realizes the app is not going to be useful, because their HCP is acting on a different set of information and therefore embarks on a different diagnostic and treatment plan than what the app recommends, rendering the app useless [36].

To make it easier for HCPs to recommend an app to patients, the HCP needs to know that the app is scientifically accurate and follows current guidelines and regulations [37]. Ideally, the app should be proven to deliver the benefits that patients desire and HCPs want to provide, however, this may be a much bigger ask than is currently possible. Our ability to evaluate apps as fast as they can change is

woefully inadequate [38]. Current approaches used to evaluate drugs do not work for evaluating mhealth apps because, while drug formulations stay relatively static, mhealth apps are constantly changing.
