The app features that are linked to the quality scores and user ratings of mental health apps: a machine learning approach

Introduction: There has been a rapid increase in demand for mental health services (in part due to COVID-19). Currently, there are over 10,000 mental health apps available in the app stores. The Organisation for the Review of Care and Health Applications (ORCHA) is a United Kingdom based digital health compliance company that specialises in the assessment of digital health apps’ quality (quality defined as “compliance with best practice standards”). For mental health apps to be recommended to the users by health professionals, they must be of sufficient quality, and some features such as ‘goal setting and gamification’ may be linked to better quality mental health apps. The objective of this study was to identify the common features within mental health apps and to determine which features are linked to quality scores and user ratings.

Methods: R language and R studio has been used to conduct inferential statistics and generate results. Random forest and Boruta feature selection methods have been used to identify the app features that are important for predicting user ratings as well as ORCHA quality scores in the areas of user experience, professional/clinical assurance, data privacy and overall aggregate ORCHA quality score. All scores are on 0–100 point scale and user rating from 1–5 star scale. Shapiro-Wilk test was used to check for normality and Wilcoxon rank sum test was used to compare scores’ distributions with P-value<.05 considered statistically significant.

Results: Out of 552 mental health apps (116 apps with both Android and iOS versions counted), the five most common features were: ‘Information provision’ (n= 548), ‘Data capture’ (n=546), ‘Data sharing’ (n=528), ‘Health monitoring’ (n=274) and ‘Goal setting & gamification’ (n=230). ‘Service signposting’ (n=221) followed by ‘Goal setting & gamification’ has been identified as the most important features for predicting the ORCHA score and user rating, according to random forest and Boruta feature selection methods. ORCHA score for 552 and user rating for 479 (unrated apps removed) mental health apps was not normally distributed. Apps that had ‘Service signposting’ or ‘Goal setting & gamification’ feature had higher ORCHA scores than those without (median increase of 8 points and 8.5 points respectively). Wilcoxon rank sum test confirms that the difference in distribution is statistically significant for both P<.001. For user rating (n=479) apps that had ‘Service signposting’ feature had a decrease of .11 stars and P-value of .002 while ‘Goal setting & gamification’ increase of .06 stars with P-value of .20.

Conclusions: This study shows that the ‘Service signposting’ and ‘Goal setting & gamification’ features are positively linked to ORCHA score. For user rating feature ‘Service signposting’ has been negatively linked to ORCHA score, achieving statistical significance and ‘Goal setting & gamification’ was positively linked to ORCHA score, but didn’t achieve statistical significance. Nevertheless, the use of ‘Service signposting’ and ‘Goal setting & gamification’ features are encouraged for mental health apps due to improvements in overall quality.