**1.10 Ecological momentary assessments and interventions are useful theoretical constructs**

There's a lot of hype about using sensors embedded in smartphones to detect changes in usage patterns which imply a worsening of disease [60] or for detection of disease [61–63]. Even very recent studies are exploratory and need additional validation before being translated into the field [64, 65]. Although the research base that makes production use of smartphone sensors is still in its infancy [66], there are some promising developments in creating architectures that are ready for conducting research. Digital (behavior) phenotyping, the ability to use wearables and sensor data to detect different behaviors, is developing rapidly and it is likely that the basic science will become more easily accessible over the next several years [67]. The developers of the geographically explicit ecological momentary assessment (GEMA) architecture claim that their approach is scalable from their study in post-partum mothers to other disease areas [68].

The use of validated patient reported measures questionnaires for conducting ecological momentary assessments is well developed and can be used with good design in terms of frequency and response burden [69, 70]. Patient reported experience and outcomes measures (PREMS and PROMS) have been developed in a variety of disease areas and are a promising way for mhealth app developers to get a jump start on ecological momentary assessments [71–74]. Use of PREMS and PROMS may incur a license fee but are worth the cost since the tools are generally validated and are widely used in clinical trials. Care must be taken to select PREMS and PROMs properly, since some of them are more useful as aggregate measures (suitable for clinical trials) but are not appropriate for use on individual patients in a clinical setting [75].

Ecological momentary interventions are another promising concept, especially in the mental health field, where cognitive and affective interventions can be delivered directly through an app [76–78]. Although it is possible for apps to be used for other interventions, such as driving insulin pumps in response to readings from a continuous glucose monitoring system [79] this is still unusual and very much in its infancy.

Ecological momentary interventions are also very promising for delivering behavior change interventions at just the right time [80]. Ideally, delivery of interventions should be timed to when people are most receptive to receiving them. Timing of interventions can be predicted by use of ecological momentary assessments to know when a patient might be at their worst or at their best and deliver an intervention at just the right time [81, 82].

#### **1.11 Effectiveness assessment requires a rapid research infrastructure and new research methodologies**

Many app guidelines request app developers to test their app for clinical benefits [13, 83]. However, testing apps can be a fraught process, because typically, apps are designed and developed using an agile process, but clinical evaluation requires a more waterfall process. The methodologies for development and evaluation are somewhat incompatible and creates friction for good quality clinical trials. Philpott et al. make some recommendations about how to create a clinical trials infrastructure that is more flexible to the needs of mobile health app evaluation. They recommend a 2-phase approach. The first phase is a qualitative phase with usability and expert reviews to ensure that the app meets minimum criteria for more rigorous testing. The second phase then evaluates the app in an adaptive randomized controlled trial, which allows low quality apps to be identified and quickly removed from the study and allows high quality apps to be tested more rigorously [38]. This ensures that scarce resources are not wasted on apps that do not add value.

## **2. Use case: myasthenia gravis**

Myasthenia gravis (MG) is a chronic autoimmune disease that targets the neuromuscular junction and causes unpredictable, fluctuating muscle weakness. Symptoms include drooping eyelids (the classical, textbook symptom), difficulty breathing, upper and lower limb weakness, difficulty swallowing and slurred speech [84]. MG affects about 20 people in every 100,000 [85]. It is twice more prevalent in women who get it in their 20s–30s than in men who get it in their 60s–80s. At one time, the mortality rate for MG was as high as 75%. However, with advances in treatment, the mortality rate is down to less than 5% [85]. Patients with MG are treated by neurologists.

We use myasthenia gravis as the use case for demonstrating a framework for designing the clinical aspects of a mobile health app.

#### **2.1 Identifying and designing for disease-specific needs**

Patient focus groups were formed before any app development activities were started to get patient input on whether an app would add value and if so, what the app should encompass. Patients agreed that an app could fill a gap that exists in care. They provided information about their experience of their disease and provided high level guidance about the features, functionality and benefits they would like to see in an app.

Clinician (neurologist) key informant interviews (N=8) were held to gather clinician experiences and gather clinician input into what they would like to see in an app for MG. Clinicians indicated that they struggle with capturing important information about the patient's disease progression, medication use and compliance and the broad range of quality-of-life issues that patients with MG face. Clinicians are also interested in learning more about objective physical signs and not just the subjective patient experience. Key clinician-anticipated benefits from an app include getting a better, more detailed timeline of key events without the fog of recall bias and getting a better sense of medication compliance. There was no consensus on how to track patients between visits, but everyone knew about the MG Activities of Daily Living questionnaire (MG-ADL) [86]. Clinicians indicated that an app would also benefit primary care physicians, who are taking on more of the care of patients with MG.

Key neurology opinion leaders (N=7) were also interviewed to better understand the context in which the app would be introduced, what metrics the app would need to exhibit for them to support it and reasons they would need to recommend it to other clinicians.
