Using data science techniques to analyse quality assessment data of digital health apps

Abstract

Digital health apps are a popular supplement to traditional healthcare. There are more than 350,000 digital health apps available in the app stores. To mitigate the risks associated with their use, they must be assessed for quality. The Organization for the Review of Care and Health Apps (referred to as ORCHA) is a United Kingdom based company that specialises in the quality assessment of digital health apps. A key objective of this thesis was to increase our understanding of the quality of digital health apps by undertaking secondary data analytics and machine learning analysis of a unique and novel ORCHA dataset.

In this PhD, methods such as meta-analysis and a systematic search of literature were used. Quantitative methods such as data science and machine learning techniques, including descriptive statistics, cluster analysis, and multiple linear regression were also used to help discover useful and actionable insights into the quality of health apps. The thesis contains an umbrella review of current literature (15 review articles) regarding the assessment criteria required to effectively assess the quality of digital health apps. A meta-analysis of 114 digital health apps was carried out to confirm whether a widely accepted system usability score (SUS) distribution and threshold (i.e. the accepted mean SUS score of 68 with a standard deviation of 12.5) is appropriate for assessing the usability of health apps. The analysis confirmed that a SUS threshold of a mean of 68 and standard deviation of 12.5 is suitable to use in benchmarking the usability of digital health apps. However, when testing for normality, the Shapiro-Wilk test for physical activity apps had P=.001, indicating that SUS scores for those apps were not normally distributed as opposed to other categories combined.

Most of the studies during the PhD were conducted using a dataset provided by ORCHA. The ORCHA dataset was generated from the company’s use of their own assessment tool, referred to as the ORCHA baseline review, which is used to assess the quality of digital health apps. The datasets contained over 2000 assessments of digital health apps. The majority of the digital health apps (1542 out of 1584, 97.3%) that have been assessed in this PhD did not comply with any user experience, usability or accessibility guidelines. Moreover, 57.3% of digital health apps (902 out of 1574) had an ORCHA score below the accepted ORCHA threshold of 65. Apps that fall under the categories of respiratory and urology often tended to score the highest, and apps under the allergy and ophthalmology categories tended to score the lowest when using the ORCHA baseline review. This PhD also discovered that digital health apps can be assigned to four clusters, specifically, 1) Apps with poor user ratings (220 out of 1402, 15.7%), 2) Apps with poor professional/clinical assurance and data privacy (252 out of 1402, 18.0%), 3) Apps with poor professional/clinical assurance (415 out of 1402, 29.6%) and 4) Higher quality apps with higher user ratings (515 out of 1402, 36.7%). Another study in this PhD discovered that the quality of digital health apps is not linked to the apps’ user ratings or the number of downloads they get in the app stores.

Date of Award	May 2025
Original language	English
Supervisor	Jorge Martinez Carracedo (Supervisor), Maurice Mulvenna (Supervisor) & Raymond Bond (Supervisor)