Prediction and Prevention of Non-Adherence to Digital Health Interventions

Digital health interventions (DHIs) show vast potential in supporting patients and health care systems with the globally increasing prevalence and economic costs of non-communicable diseases (NCDs) – the leading causes of death and disability worldwide [1, 2]. More specifically, Mobile Health Applications (mHealth apps) are now considered an accessible and scalable solution to promote behavior change among patients, improve health outcomes, and reduce healthcare costs [3-5]. Correspondingly, the number of available mHealth apps has continuously grown to more than 300.000, with approximately 200 new mHealth apps released each day [2, 6].

However, despite increasing evidence and availability, mHealth apps are subject to significant dropout rates, with a substantial proportion of users not adhering to them as intended [7, 8]. Recent research has shown that up to 80 percent of all participants in mHealth interventions only engage at a minimum level, not logging into the mHealth app more than once and not consistently using the app long term [9]. Studies examining mHealth app usage in more extensive real-world settings further show that less than five percent of participants are daily active users, with the large majority only downloading the corresponding app but not using it regularly [10, 11].

Since non-adherence relative to intended use jeopardizes treatment success and thus might lead to an increased number of hospitalizations, it is considered a fundamental concern in developing mHealth apps [8, 12-15]. Still, the scientific body of literature lacks concise conceptualizations and measures for intended use of mHealth apps, while intervention components and other factors influencing adherence remain to be explored [13, 16].

Therefore, understanding factors that act as barriers or facilitators to adherence is crucial to prevent intervention dropouts and increase the effectiveness of digital health interventions. Furthermore, identifying users at risk and preventing them from dropping out indicates the potential to increase the efficiency and quality of DHIs as many interventions are only beneficial if people use them regularly and for a prolonged period [14, 15, 17-19].

Research Objectives

Predicting and preventing intervention dropouts and thus increasing the effectiveness of mHealth apps requires multiple steps: First, understanding the factors that act as barriers or facilitators to user adherence. Second, being able to predict dropouts based on individual real-time data. Third, incorporating effective preventive intervention components and actions that induce behavior change at the right time before the dropout occurs. To this end, we will use our technical expertise in the state of receptivity to fill this research gap and apply it to the prediction and prevention of non-adherence [20, 21].

In this regard, our project has the following objectives:

Identifying factors that influence adherence and predict dropouts based on existing literature and real-world data from longitudinal studies of different health domains.
Identifying intervention components that effectively intervene in critical situations to prevent dropout
Developing and evaluating an early warning system that can predict and prevent dropouts.

From a practical perspective, these objectives aim to help stakeholders of DHIs in developing more compelling and effective mHealth apps.

Planned Studies and Research Questions

We are planning three distinct studies which combine evidence from the existing body of research, retrospective analysis of longitudinal data from real-world mHealth apps, and prospective data analysis to validate our findings.

Study 1: Systematic Literature Review

Conducted in cooperation with the Swiss Institute for Addiction and Health Research (ISGF) and funded by the Swiss Federal Office of Public Health, Study 1 aimed to identify intervention- and patient-related factors influencing adherence to the intended use of mHealth apps. We further derived quantified adherence scores for different health domains, which may serve as adherence benchmarks.

A comprehensive systematic literature search (January 2007- December 2020) was conducted in MEDLINE, Embase, Web of Science, Scopus, and ACM Digital Library. Data on intended use, actual use, and factors influencing adherence were extracted. Intervention-related and patient-related factors with a positive or negative influence on app use are presented separately for the health domains NCD-Self-Management, Mental Health, Substance Use, Nutrition, Physical Activity, Weight Loss, Multicomponent Lifestyle Interventions, Mindfulness, and other NCDs. Quantified measures of adherence were derived for each study and compared with qualitative findings.

The following research questions are addressed in Study 1:

RQ 1.1: Which intervention-related factors influence adherence relative to the intended use of mHealth applications targeting NCDs in adults?
RQ 1.2: Which patient-related factors influence adherence relative to the intended use of mHealth applications targeting NCDs in adults?
RQ 1.3: How do adherence rates of mHealth apps for NCDs compare across different health domains?

The results of Study 1 can be found in the corresponding publication which is accessible via the following link: https://jmir.org/2022/5/e35371 [31]

Study 2: Retrospective Data Analysis

In study 2, we strive to identify variables (e.g., login rates, session length, or interaction frequency with specific features) and methods (e.g., logistic regression, decision trees, or random forest models) that effectively predict dropouts in mHealth apps. We also aim to identify intervention components affecting intervention adherence during the process, thus validating and potentially adding to the findings from Study 1. Furthermore, we are going to visualize cohort retention rates and compare them across different mHealth apps, which may grant further insights into common points of dropouts. For this purpose, we collaborate with the following industry project partners:

Manoa – Pathmate Technologies AG (Website)

The digital coach Manoa supports self-managing high blood pressure, type 2 diabetes, and sleep problems in everyday life and motivates users to reduce influenceable risk factors. The app was developed with medical experts and supports the correct documentation of relevant data (e.g., blood pressure, blood glucose, sleep), calculates average values, and provides feedback based on the current state of scientific knowledge and medical guidelines. Manoa is the name of the Pathmate Coach, which is registered as a medical device class I.

Ready4life – Swiss Institute for Addiction and Health Research – ISGF (Website)

The lifestyle intervention ready4life promotes coping with stress, strengthening social skills, and resisting addictive substances. Targeting young people from 15 to 25 years, the app is aimed at learners in vocational schools, polytechnic schools, and training companies in Germany, Liechtenstein, Switzerland, and Austria. Within the app, users can choose a coach (avatar) to guide them through the program.

MySwissFoodPyramid – the Federal Food Safety and Veterinary Office – BLV (Website)

The nutrition app “MySwissFoodPyramid” aims at increasing food literacy in the Swiss population by an interactive food diary and personalized feedback through the MobileCoach-based chat interface and the digital coaches Lukas or Anna. The app is based on Swiss dietary recommendations and aims at people aged 16 and older. The Federal Food Safety and Veterinary Office (BLV) developed the app in collaboration with Pathmate Technologies.

lvlUP – Singapore ETH-Centre (Website)

The holistic lifestyle interbention “lvlUP” aims at supporting adolescents in Singapore with physical exercise, stress management, and healthy eating habits. LvL UP includes several tools, including chatbot coaching, a gameful slow-paced breathing game, and journaling.

WayBetter – WayBetter, Inc. (Website)

WayBetter is a weight management intervention utilizing gamification to motivate users to build new healthy eating and exercise habits, helping users stay accountable to sticking to new habits, and providing users with a supportive community. So far, 1.3 million players have already lost 14.7 million pounds using WayBetter.

To find effective variables and methods to predict dropouts within these interventions, we will explore a variety of machine learning techniques that have been successfully applied for customer churn prediction in healthcare [14, 17-19], but also other areas such as telecommunications [22-24] or gaming [25-29].

In summary, Study 2 addresses the following research questions:

RQ 2.1: Which variables predict dropouts in mHealth apps? (e.g., login rates, session length, or interaction frequency with specific features)
RQ 2.2: Which methods effectively predict dropouts of patients in mHealth apps? (e.g., logistic regression, decision trees, or random forest models

Study 3: Prospective Analysis and Outcome Validation

Applying findings from Studies 1 & 2, we want to develop and evaluate a prototype of an early warning system that predicts and prevents intervention dropouts. As part of Study 3, we plan to assess the efficacy of this tool with one of our project partners.

In a first step, the tool is supposed to predict dropouts processing adequate real-time socio-demographic, medical, and usage data derived from Studies 1 & 2 applying methods derived from Study 2. In a second step, once a dropout is predicted for a specific user, the tool triggers a preventive action based on findings from Studies 1 & 2. The tool’s efficacy will be evaluated by comparing dropout predictions with actual dropout data and comparing dropout rates of the intervention group (receiving preventive action) with dropout rates of a control group.

The outcome validation of Study 3 aims to answer the following research question:

RQ 3: Are selected actions effective in preventing dropouts?

Results and Possible Real-World Implementations

Preliminary Results

The literature search for Study 1 yielded 2862 potentially relevant articles, of which 99 were included as part of the inclusion criteria. Four intervention-related factors indicated positive effects on adherence across all health domains: (1) personalization or tailoring the content of the mHealth app to the individual needs of the user, (2) reminders in the form of individualized push notifications, (3) a user-friendly app design, and technical stability, and (4) personal support complementary to the digital intervention. Social and gamification features were also identified as drivers of app adherence across several health domains. A wide variety of patient-related factors like socio-demographic characteristics or type of user acquisition channel further affected adherence. Derived adherence scores of included mHealth apps averaged 56.0% and were highest for Multicomponent Lifestyle Interventions (60.1%) and lowest for apps aimed at reducing substance use (46.1%).

From a practical perspective, the identified intervention-related factors can be used by mHealth app developers as Feature, UI, and UX development guidelines. The identified intervention components positively influencing adherence serve, in combination with findings from Study 2, as an inspiration for potential preventive actions in Study 3. The derived adherence scores across different health apps may also serve as benchmarks to evaluate mHealth apps and draw further inspiration from relatively high-performing apps within their specific health domain. Finally, the identified patient-related factors reveal potential confounding factors when evaluating apps regarding adherence or predicting dropouts for specific populations. These identified factors are now subject to validation in Study 2 and serve to develop preventive actions in Study 3.

Study 1 also confirmed previous findings by Sieverink et al. (2017), that “adherence to eHealth technology is an underdeveloped and often improperly used concept in the existing body of literature” [13] and that “justifications for intended use are often missing in evaluations of adherence” [13]. Deriving intended use of 97 individual mHealth apps in Study 1 showed that intended use differs across mHealth apps with daily tracking (e.g., daily diary entries, 36/97, 36.1%), activity completion (e.g., completion of a certain number of coaching modules, 19/97, 19.6%), and daily usage (e.g., daily login, 17/97, 17.5%) being the most common categories of intended use

Expected Results

The results of Study 1 suggest that adherence must be defined for each intervention individually, as mHealth apps vary significantly in terms of intended use, intended interaction frequency, and intended interaction duration. Still, we will continue to explore how adherence can be standardized across different mHealth apps, also reflecting on how adherence is measured in non-digital medical treatment and how it developed over time. One extreme measure of adherence with potential universal application for DHIs is the dropout rate, defined as the percentage of users who entirely cease to use the intervention during the intervention period, which closely relates to the term “customer churn” established in other app industries [22-29].

Retrospectively analyzing longitudinal data from mHealth applications in Study 2, we want to identify intervention components affecting intervention adherence, thus validating and potentially adding to the findings from Study 1. We also aim to visualize retention curves and compare them across different mHealth apps to identify common points of dropouts. Most importantly, we strive to identify effective variables and methods predicting intervention dropouts. If identified variables and methods do not predict or explain dropouts with considerable sensitivity and specificity, we can include more features or data sets from additional mHealth apps in the model. Combining the results from Study 2, we expect to predict potential dropouts and identify ideal situations for preventive actions in real-time for the studied interventions.

Based on the results from Studies 1 & 2, we strive to develop a software tool capable of predicting and preventing dropouts in mHealth apps. Evaluating the tool in Study 3, we aim to measure the degree of beneficial effect under real-world clinical settings. This process will grant insights into the extent to which dropouts can be predicted and if selected actions effectively prevent intervention dropouts. In the best case, the model can predict dropouts with good sensitivity and specificity, while the dropout rate is significantly lower in the intervention group than the control group. Regarding the design of the preventive actions, we aim to minimize the potentially harmful effects of type 1 errors (False positive), which will also determine the underlying measures to evaluate the model’s predictive performance.

Potential real-world applications based on these findings could be the development of a SaaS solution that supports mHealth app developers in predicting and preventing intervention dropouts. Such a service might also extend the scope of mHealth interventions with potential applications for other app industries, particularly for subscription-based services.

Looking for a Master thesis?

Dropout Prediction in Digital Health Interventions

Candidate profile: We are searching for highly motivated, proactive, and self-directed students with a general interest in digital health. You should be proficient in creating clear and comprehensible Python scripts that can be reused by other parties to explore and analyze large datasets. You should have experience cleaning and organizing data and applying relevant statistical models. You will be fully supported as to relevant theory from medical and behavioral sciences and how to write your thesis for a healthcare audience.
Master thesis tasks: You will clean, organize, and analyze given data sets, conduct statistical analysis, engineer and select feature variables, and explore machine learning methods that classify potential dropouts in mHealth apps. Your work will be part of a more extensive scientific study which is subject to being published in a renowned medical peer-reviewed journal, which you are invited to co-author based on your contribution. The project can be conducted entirely remotely. We highly encourage members of underrepresented groups to apply. Students from other universities than ETH Zurich are also welcome to apply.
Start: Anytime, as soon as possible
Methodology: Longitudinal data analysis
Duration: 6 months (or to be discussed)
Interested students are invited to send an email to Robert Jakob at rjakob@ethz.ch.

Systematic Review of User Churn Prediction Literature

Candidate profile: We are searching for highly motivated, proactive, and self-directed students with a strong interest in applying machine learning techniques to predict user churn in mobile apps and a desire to conduct a comprehensive systematic review of the existing literature in this area. The ideal candidate will have a strong background in data science and a desire to use data to drive decision-making and improve user engagement with mobile apps.
Master thesis/internship tasks: You will conduct a systematic review of the existing literature on user churn prediction with mobile app data. Your work will be part of a more extensive scientific study that is subject to being published in a renowned medical peer-reviewed journal, which you are invited to co-author based on your contribution. The project can be conducted entirely remotely. We highly encourage members of underrepresented groups to apply. Students from other universities than ETH Zurich are also welcome to apply.
Start: Anytime, as soon as possible
Methodology: Longitudinal data analysis
Duration: 6 months (or to be discussed)
Interested students are invited to send an email to Robert Jakob at rjakob@ethz.ch.

Systematic Review of Engagement Parameters utilized across Digital Health Studies

Candidate profile: We are searching for highly motivated, proactive, and self-directed students with a strong interest in health research methods, data analysis and visualization. Familiarity with digital health studies and engagement parameters is desirable. The ideal candidate will have experience in working with health-related data, conducting systematic reviews, and writing research papers.
Master thesis/internship tasks: You will conduct a systematic review of how engagement parameters (e.g., Retention Rate, Attrition Rate, Adherence, etc.) are calculated and reported across digital health studies. Your work will be part of a more extensive scientific study that is subject to being published in a renowned medical peer-reviewed journal, which you are invited to co-author based on your contribution. The project can be conducted entirely remotely. We highly encourage members of underrepresented groups to apply. Students from other universities than ETH Zurich are also welcome to apply.
Start: Anytime, as soon as possible
Methodology: Longitudinal data analysis
Duration: 6 months (or to be discussed)
Interested students are invited to send an email to Robert Jakob at rjakob@ethz.ch.

Publications

Jakob R, Harperink S, Rudolf A, Fleisch E, Haug S, Mair J, Salamanca-Sanabria A, Kowatsch T. 2022. Factors Influencing Adherence to mHealth Apps for Prevention or Management of Noncommunicable Diseases: Systematic Review. J Med Internet Res 2022;24(5):e35371. URL: https://www.jmir.org/2022/5/e35371. DOI: 10.2196/35371

Paz Castro R, Haug S, Debelak R, Jakob R, Kowatsch T, Schaub M. Engagement With a Mobile Phone–Based Life Skills Intervention for Adolescents and Its Association With Participant Characteristics and Outcomes: Tree-Based Analysis. J Med Internet Res 2022;24(1):e28638. DOI:10.2196/28638

Federal Office of Public Health of the Swiss Confederation (BAG). Faktenblatt: Erfolgsfaktoren von mHealth-Applikationen. 2022. URL: https://www.prevention.ch/article/erfolgsfaktoren-von-mhealth-applikationen

Haug S, Augsburger M, Jakob R, Kowatsch T. Literaturstudie zu Verhaltensänderungen durch mHealth Applikationen. 2021. Federal Office of Public Health of the Swiss Confederation (BAG). DOI: 10.3929/ethz-b-000549326.

Related Work

WHO. Noncommunicable diseases. 2021.
Suisse e. eHealth Suisse. 2020.
Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol. 2015;6:27882. PMID: 25994025. doi: 10.3402/ejpt.v6.27882.
Wang Y, Xue H, Huang Y, Huang L, Zhang D. A Systematic Review of Application and Effectiveness of mHealth Interventions for Obesity and Diabetes Treatment and Self-Management. Adv Nutr. 2017 May;8(3):449-62. PMID: 28507010. doi: 10.3945/an.116.014100.
Shaw J, Agarwal P, Desveaux L, Palma DC, Stamenova V, Jamieson T, et al. Beyond “implementation”: digital health innovation and service design. NPJ Digit Med. 2018;1:48. PMID: 31304327. doi: 10.1038/s41746-018-0059-8.
Aitken M, Clancy, B., & Nass, D. The Growing Value of Digital Health – Evidence and Impact on Human Health and the Healthcare System. 2017.
Hesser H. Estimating causal effects of internet interventions in the context of nonadherence. Internet Interv. 2020 Sep;21:100346. PMID: 32983907. doi: 10.1016/j.invent.2020.100346.
Eysenbach G. The law of attrition. J Med Internet Res. 2005 Mar 31;7(1):e11. PMID: 15829473. doi: 10.2196/jmir.7.1.e11.
Meyerowitz-Katz G, Ravi S, Arnolda L, Feng X, Maberly G, Astell-Burt T. Rates of Attrition and Dropout in App-Based Interventions for Chronic Disease: Systematic Review and Meta-Analysis. J Med Internet Res. 2020 Sep 29;22(9):e20283. PMID: 32990635. doi: 10.2196/20283.
Linardon J, Fuller-Tyszkiewicz M. Attrition and adherence in smartphone-delivered interventions for mental health problems: A systematic and meta-analytic review. J Consult Clin Psychol. 2020 Jan;88(1):1-13. PMID: 31697093. doi: 10.1037/ccp0000459.
Baumel A, Muench F, Edan S, Kane JM. Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. J Med Internet Res. 2019 Sep 25;21(9):e14567. PMID: 31573916. doi: 10.2196/14567.
Morrissey EC, Corbett TK, Walsh JC, Molloy GJ. Behavior Change Techniques in Apps for Medication Adherence: A Content Analysis. Am J Prev Med. 2016 May;50(5):e143-e6. PMID: 26597504. doi: 10.1016/j.amepre.2015.09.034.
Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence. J Med Internet Res. 2017 Dec 6;19(12):e402. PMID: 29212630. doi: 10.2196/jmir.8578.
Pedersen DH, Mansourvar M, Sortsø C, Schmidt T. Predicting Dropouts From an Electronic Health Platform for Lifestyle Interventions: Analysis of Methods and Predictors. J Med Internet Res. 2019;21(9):e13617. PMID: 31486409. doi: 10.2196/13617.
Trinh H, Shamekhi A, Kimani E, Bickmore TW. Predicting User Engagement in Longitudinal Interventions with Virtual Agents. Proceedings of the 18th International Conference on Intelligent Virtual Agents; Sydney, NSW, Australia: Association for Computing Machinery; 2018. p. 9–16.
Baumel A, Yom-Tov E. Predicting user adherence to behavioral eHealth interventions in the real world: examining which aspects of intervention design matter most. Transl Behav Med. 2018 Sep 8;8(5):793-8. PMID: 29471424. doi: 10.1093/tbm/ibx037.
Bremer V, Chow PI, Funk B, Thorndike FP, Ritterband LM. Developing a Process for the Analysis of User Journeys and the Prediction of Dropout in Digital Health Interventions: Machine Learning Approach. J Med Internet Res. 2020 Oct 28;22(10):e17738. PMID: 33112241. doi: 10.2196/17738.
Kwon H, Kim HH, An J, Lee JH, Park YR. Lifelog Data-Based Prediction Model of Digital Health Care App Customer Churn: Retrospective Observational Study. J Med Internet Res. 2021 Jan 6;23(1):e22184. PMID: 33404511. doi: 10.2196/22184.
SARINE BABIKIAN VS, MARK A. CLEMENTS. 862-P: A Machine Learning Model Predicts Engagement with a Mobile Health (mHealth)-Based Remote Patient Monitoring (RPM) Program among Persons with Diabetes (PWDs). Diabetes Jun 2020, 69 (Supplement 1) 862-P. 2020. doi: 10.2337/db20-862-P.
Kramer JN, Kunzler F, Mishra V, Presset B, Kotz D, Smith S, et al. Investigating Intervention Components and Exploring States of Receptivity for a Smartphone App to Promote Physical Activity: Protocol of a Microrandomized Trial. JMIR Res Protoc. 2019 Jan 31;8(1):e11540. PMID: 30702430. doi: 10.2196/11540.
Künzler F, Mishra V, Kramer J-N, Kotz D, Fleisch E, Kowatsch T. Exploring the State-of-Receptivity for mHealth Interventions. Proc ACM Interact Mob Wearable Ubiquitous Technol. 2019;3(4):Article 140. doi: 10.1145/3369805.
Raja B, Jeyakumar P. An Effective Classifier for Predicting Churn in Telecommunication. Journal of Advanced Research in Dynamical and Control Systems. 2019 06/07;11:221-9.
Vafeiadis T, Diamantaras KI, Sarigiannidis G, Chatzisavvas KC. A comparison of machine learning techniques for customer churn prediction. Simulation Modelling Practice and Theory. 2015 2015/06/01/;55:1-9. doi: https://doi.org/10.1016/j.simpat.2015.03.003.
Ahmad AK, Jafar A, Aljoumaa K. Customer churn prediction in telecom using machine learning in big data platform. Journal of Big Data. 2019 2019/03/20;6(1):28. doi: 10.1186/s40537-019-0191-6.
Kawale J, Pal A, Srivastava J, editors. Churn Prediction in MMORPGs: A Social Influence Based Approach. 2009 International Conference on Computational Science and Engineering; 2009 29-31 Aug. 2009.
Hadiji F, Sifa R, Drachen A, Thurau C, Kersting K, Bauckhage C, editors. Predicting player churn in the wild. 2014 IEEE Conference on Computational Intelligence and Games; 2014 26-29 Aug. 2014.
Á P, Saas A, Guitart A, Magne C, editors. Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using Survival Ensembles. 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA); 2016 17-19 Oct. 2016.
Bertens P, Guitart A, Á P, editors. Games and big data: A scalable multi-dimensional churn prediction model. 2017 IEEE Conference on Computational Intelligence and Games (CIG); 2017 22-25 Aug. 2017.
Runge J, Gao P, Garcin F, Faltings B, editors. Churn prediction for high-value players in casual social games. 2014 IEEE Conference on Computational Intelligence and Games; 2014 26-29 Aug. 2014.
Mishra H, Enami S, Nielsen RJ, Stewart LA, Hoffmann MR, Goddard WA, 3rd, et al. Bronsted basicity of the air-water interface. Proc Natl Acad Sci U S A. 2012 Nov 13;109(46):18679-83. PMID: 23112167. doi: 10.1073/pnas.1209307109.
Jakob R, Harperink S, Rudolf AM, Fleisch E, Haug S, Mair JL, Salamanca-Sanabria A, Kowatsch T. Factors Influencing Adherence to mHealth Apps for Prevention or Management of Noncommunicable Diseases: Systematic Review. J Med Internet Res 2022;24(5):e35371. doi: 10.2196/35371 . PMID: 35612886