Usability analysis of a mental health and wellbeing chatbot, and the association between usability and perceived effectiveness

In this study we examine the usability of a mental wellbeing chatbot called ChatPal, which was developed in collaboration between universities and mental health charities based on the concept of positive psychology. Volunteers were recruited to trial the chatbot over a period of twelve weeks and were then requested to
complete a survey on the usability of the application. The survey consisted of twelve statements with responses using the Likert scale with a free text area for other feedback. A total of 131 completed surveys were recorded and analysis of the results show that while 38% of participants found the chatbot realistic and engaging, 36% did not. When questioned regarding whether they felt the chatbot was too robotic, 43% agreed with this characterisation while 33% did not agree. Almost half felt that
the chatbot was not welcoming during the initial set up with 54% indicating that they felt that the bot seemed unfriendly whilst interacting with it. Opinions were also split on understanding the purpose of the chatbot with 48% indicating that it did not explain its scope and purpose well (40% felt that it did) and 53% indicating that it
gave no indication as to its purpose (38% disagreed). Even though 40% felt that it was easy to navigate the menu system, the same proportion of participants felt that it was easy to get confused when using it. A quarter of participants felt that not only could the chatbot not cope well with errors, but it was unable to handle them, which likely points to the challenge of handling free text. When questioned on how useful they thought the chatbot was for supporting mental wellbeing, over 74% felt that it was useful and only 15% disagreed. As for how much they believed the chatbot had changed their mental wellbeing, over 54% felt that it had improved their mental well-being with almost 6% feeling that it had a detrimental effect. A strong correlation was found between how useful participants felt the chatbot was for supporting mental health and their perception of the improvement in their own mental well-being ( r=.74). The results show a split in opinions as to the usability of the ChatPal chatbot. The results indicate the challenge of designing the chatbot’s 'personality’, purpose and ease of navigation. Studies have shown that chatbots exhibiting high levels of conversational skill are more likely to be adopted by users than those with lower skill levels. People perceive a more skilled chatbot to be more socially present and anthropomorphic than a less skilled one. As ChatPal is essentially a menu driven chatbot, it was perceived to be robotic. Despite the technical limitations of the chatbot, the majority of participants felt that it was useful for supporting their
mental wellbeing with over a half responding that it had a positive effect on it.

The authors would like to thank the clients, participants, project members, supporters, and researchers at Ulster University, University of Eastern Finland, Norrbotten Association of Local Authorities, Region Norrbotten, Luleå University of Technology, NHS Western Isles, Action Mental Health, Munster Technological University, and Health Innovation Hub Ireland for the successful development, implementation, and evaluation of this research. The ChatPal project is supported by Interreg Northern Periphery and Arctic Programme under grant number 345.