Abstract
Background:
There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.
Objective:
The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).
Methods:
We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.
Results:
We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI’s health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.
Conclusions:
There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women’s health, sexual health, and dental care may be needed.
There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.
Objective:
The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).
Methods:
We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.
Results:
We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI’s health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.
Conclusions:
There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women’s health, sexual health, and dental care may be needed.
| Original language | English |
|---|---|
| Article number | e47043 |
| Pages (from-to) | 1-13 |
| Number of pages | 13 |
| Journal | JMIR mHealth and uHealth |
| Volume | 11 |
| Issue number | 1 |
| Early online date | 23 Nov 2023 |
| DOIs | |
| Publication status | Published (in print/issue) - 23 Nov 2023 |
Bibliographical note
Publisher Copyright:© 2023 JMIR Publications. All rights reserved.
Data Availability Statement
The data sets generated during and/or analyzed during this study are available in the ORCHA repository [38]. The data from this data set (and for more DHIs) are freely available to registered users (registration is free for a trial period). However, the data in the repository are presented for individual DHIs and would need to be extracted into a suitable format for analysis.Funding
This research is done in partnership with the Organisation for the Review of Care and Health Apps (ORCHA), which is a UK-based digital health compliance company. This work is supported by a Northern Ireland Department for the Economy (DfE) Co-operative Awards in Science and Technology (CAST) award PhD scholarship. We would like to acknowledge the contribution of the many DHIs reviewers and developers who worked with ORCHA to allow for the review of DHIs and consent for their data to be used for research purposes. Without their contributions and consent, this research would not have been possible
UN SDGs
This output contributes to the following UN Sustainable Development Goals (SDGs)
-
SDG 3 Good Health and Well-being
Keywords
- mHealth assessment
- digital health
- digital health interventions
- digital health apps
- quality assessment
- quantifying DHIs
- mobile health
- ORCHA assessment
- digital health interventions scoring
- Organisation for the Review of Care and Health Apps
Fingerprint
Dive into the research topics of 'Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study'. Together they form a unique fingerprint.Student theses
-
Using data science techniques to analyse quality assessment data of digital health apps
Zych, M. M. (Author), Martinez Carracedo, J. (Supervisor), Mulvenna, M. (Supervisor) & Bond, R. (Supervisor), May 2025Student thesis: Doctoral Thesis
File
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver