Usability testing and trust analysis of a mental health and wellbeing chatbot

Kyle Boyd; Courtney Potts; RR Bond; Maurice Mulvenna; Thomas Broderick; Con Burns; Andrea Bickerdike; Michael McTear; Catrine Kostenius; Alex Vakaloudis; Indika S. A. Dhanapala; Edel Ennis; Frederick Booth

doi:10.1145/3552327.3552348

Usability testing and trust analysis of a mental health and wellbeing chatbot

Kyle Boyd
, Courtney Potts
, RR Bond
, Maurice Mulvenna
, Thomas Broderick
, Con Burns
, Andrea Bickerdike
, Michael McTear
, Catrine Kostenius
, Alex Vakaloudis
, Indika S. A. Dhanapala
, Edel Ennis
, Frederick Booth

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

18 Citations (Scopus)

230 Downloads (Pure)

Abstract

Mental health chatbots are particularly useful for those who are isolated and may have difficulty attending services or for those who are reluctant to speak to a professional. In this study, the usability and trust of a chatbot known as ’ChatPal’ has been assessed. ChatPal has been developed by an interdisciplinary team encompassing health service providers, local authorities, charities and universities to promote positive mental wellbeing among individuals in rural areas across Europe. This study employed a usability test protocol to recruit representative users to complete a set of tasks using the ChatPal chatbot. Usability issues were assessed along with trust and users’ satisfaction on the System Usability Scale and the Chatbot Usability Questionnaire. The study shows the usability issues and trust with a mental health chatbot and highlights recommendations for improvement.

Original language	English
Title of host publication	ECCE 2022 - Proceedings of the 33rd European Conference on Cognitive Ergonomics
Publisher	Association for Computing Machinery
Pages	1-8
Number of pages	8
ISBN (Electronic)	978-1-4503-9808-4
DOIs	https://doi.org/10.1145/3552327.3552348
Publication status	Published (in print/issue) - 4 Oct 2022
Event	33rd European Conference on Cognitive Ergonomics - Kaiserslautern, Germany Duration: 4 Oct 2022 → 7 Oct 2022 https://hci.uni-kl.de/ecce2022/

Conference

Conference	33rd European Conference on Cognitive Ergonomics
Abbreviated title	ECCE 2022
Country/Territory	Germany
City	Kaiserslautern
Period	4/10/22 → 7/10/22
Internet address	https://hci.uni-kl.de/ecce2022/

Bibliographical note

Funding Information:
The ChatPal consortium acknowledges the support provided by the Interreg VB Northern Periphery & Arctic Programme under the grant for Conversational Interfaces Supporting Mental Health and Wellbeing of People in Sparsely Populated Areas (ChatPal) project number 345. The authors would like to thank all the clients, participants, project members, supporters, and researchers at Ulster University, University of Eastern Finland, Norrbotten Association of Local Authorities, Region Norrbotten, Luleå University of Technology, NHS Western Isles, Action Mental Health, Munster Technological University, and Health Innovation Hub Ireland, for participating in this research.

Publisher Copyright:
© 2022 ACM.

Funding

Funding Information: The ChatPal consortium acknowledges the support provided by the Interreg VB Northern Periphery & Arctic Programme under the grant for Conversational Interfaces Supporting Mental Health and Wellbeing of People in Sparsely Populated Areas (ChatPal) project number 345. The authors would like to thank all the clients, participants, project members, supporters, and researchers at Ulster University, University of Eastern Finland, Norrbotten Association of Local Authorities, Region Norrbotten, Luleå University of Technology, NHS Western Isles, Action Mental Health, Munster Technological University, and Health Innovation Hub Ireland, for participating in this research. Publisher Copyright: © 2022 ACM.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being
SDG 9 Industry, Innovation, and Infrastructure
SDG 11 Sustainable Cities and Communities

Keywords

Chatbots
Conversational user interfaces
User experience
Digital interventions
Apps
Mental health
Wellbeing

Access to Document

10.1145/3552327.3552348Licence: CC BY-ND

ecce2022-21Final published version, 678 KBLicence: CC BY

Cite this

Boyd, K., Potts, C., Bond, RR., Mulvenna, M., Broderick, T., Burns, C., Bickerdike, A., McTear, M., Kostenius, C., Vakaloudis, A., Dhanapala, I. S. A., Ennis, E., & Booth, F. (2022). Usability testing and trust analysis of a mental health and wellbeing chatbot. In ECCE 2022 - Proceedings of the 33rd European Conference on Cognitive Ergonomics (pp. 1-8). Article 18 Association for Computing Machinery. https://doi.org/10.1145/3552327.3552348

@inproceedings{a10f8805c8d940e5b6f08a71e3d84fb6,

title = "Usability testing and trust analysis of a mental health and wellbeing chatbot",

abstract = "Mental health chatbots are particularly useful for those who are isolated and may have difficulty attending services or for those who are reluctant to speak to a professional. In this study, the usability and trust of a chatbot known as {\textquoteright}ChatPal{\textquoteright} has been assessed. ChatPal has been developed by an interdisciplinary team encompassing health service providers, local authorities, charities and universities to promote positive mental wellbeing among individuals in rural areas across Europe. This study employed a usability test protocol to recruit representative users to complete a set of tasks using the ChatPal chatbot. Usability issues were assessed along with trust and users{\textquoteright} satisfaction on the System Usability Scale and the Chatbot Usability Questionnaire. The study shows the usability issues and trust with a mental health chatbot and highlights recommendations for improvement.",

keywords = "Chatbots, Conversational user interfaces, User experience, Digital interventions, Apps, Mental health, Wellbeing",

author = "Kyle Boyd and Courtney Potts and RR Bond and Maurice Mulvenna and Thomas Broderick and Con Burns and Andrea Bickerdike and Michael McTear and Catrine Kostenius and Alex Vakaloudis and Dhanapala, \{Indika S. A.\} and Edel Ennis and Frederick Booth",

note = "Funding Information: The ChatPal consortium acknowledges the support provided by the Interreg VB Northern Periphery \& Arctic Programme under the grant for Conversational Interfaces Supporting Mental Health and Wellbeing of People in Sparsely Populated Areas (ChatPal) project number 345. The authors would like to thank all the clients, participants, project members, supporters, and researchers at Ulster University, University of Eastern Finland, Norrbotten Association of Local Authorities, Region Norrbotten, Lule{\aa} University of Technology, NHS Western Isles, Action Mental Health, Munster Technological University, and Health Innovation Hub Ireland, for participating in this research. Publisher Copyright: {\textcopyright} 2022 ACM.; 33rd European Conference on Cognitive Ergonomics, ECCE 2022 ; Conference date: 04-10-2022 Through 07-10-2022",

year = "2022",

month = oct,

day = "4",

doi = "10.1145/3552327.3552348",

language = "English",

pages = "1--8",

booktitle = "ECCE 2022 - Proceedings of the 33rd European Conference on Cognitive Ergonomics",

publisher = "Association for Computing Machinery",

address = "United States",

url = "https://hci.uni-kl.de/ecce2022/",

}

Boyd, K , Potts, C , Bond, RR , Mulvenna, M, Broderick, T, Burns, C, Bickerdike, A, McTear, M, Kostenius, C, Vakaloudis, A, Dhanapala, ISA, Ennis, E & Booth, F 2022, Usability testing and trust analysis of a mental health and wellbeing chatbot. in ECCE 2022 - Proceedings of the 33rd European Conference on Cognitive Ergonomics., 18, Association for Computing Machinery, pp. 1-8, 33rd European Conference on Cognitive Ergonomics, Kaiserslautern, Germany, 4/10/22. https://doi.org/10.1145/3552327.3552348

TY - GEN

T1 - Usability testing and trust analysis of a mental health and wellbeing chatbot

AU - Boyd, Kyle

AU - Potts, Courtney

AU - Bond, RR

AU - Mulvenna, Maurice

AU - Broderick, Thomas

AU - Burns, Con

AU - Bickerdike, Andrea

AU - McTear, Michael

AU - Kostenius, Catrine

AU - Vakaloudis, Alex

AU - Dhanapala, Indika S. A.

AU - Ennis, Edel

AU - Booth, Frederick

N1 - Funding Information: The ChatPal consortium acknowledges the support provided by the Interreg VB Northern Periphery & Arctic Programme under the grant for Conversational Interfaces Supporting Mental Health and Wellbeing of People in Sparsely Populated Areas (ChatPal) project number 345. The authors would like to thank all the clients, participants, project members, supporters, and researchers at Ulster University, University of Eastern Finland, Norrbotten Association of Local Authorities, Region Norrbotten, Luleå University of Technology, NHS Western Isles, Action Mental Health, Munster Technological University, and Health Innovation Hub Ireland, for participating in this research. Publisher Copyright: © 2022 ACM.

PY - 2022/10/4

Y1 - 2022/10/4

N2 - Mental health chatbots are particularly useful for those who are isolated and may have difficulty attending services or for those who are reluctant to speak to a professional. In this study, the usability and trust of a chatbot known as ’ChatPal’ has been assessed. ChatPal has been developed by an interdisciplinary team encompassing health service providers, local authorities, charities and universities to promote positive mental wellbeing among individuals in rural areas across Europe. This study employed a usability test protocol to recruit representative users to complete a set of tasks using the ChatPal chatbot. Usability issues were assessed along with trust and users’ satisfaction on the System Usability Scale and the Chatbot Usability Questionnaire. The study shows the usability issues and trust with a mental health chatbot and highlights recommendations for improvement.

AB - Mental health chatbots are particularly useful for those who are isolated and may have difficulty attending services or for those who are reluctant to speak to a professional. In this study, the usability and trust of a chatbot known as ’ChatPal’ has been assessed. ChatPal has been developed by an interdisciplinary team encompassing health service providers, local authorities, charities and universities to promote positive mental wellbeing among individuals in rural areas across Europe. This study employed a usability test protocol to recruit representative users to complete a set of tasks using the ChatPal chatbot. Usability issues were assessed along with trust and users’ satisfaction on the System Usability Scale and the Chatbot Usability Questionnaire. The study shows the usability issues and trust with a mental health chatbot and highlights recommendations for improvement.

KW - Chatbots

KW - Conversational user interfaces

KW - User experience

KW - Digital interventions

KW - Apps

KW - Mental health

KW - Wellbeing

UR - https://www.scopus.com/pages/publications/85139957448

U2 - 10.1145/3552327.3552348

DO - 10.1145/3552327.3552348

M3 - Conference contribution

SP - 1

EP - 8

BT - ECCE 2022 - Proceedings of the 33rd European Conference on Cognitive Ergonomics

PB - Association for Computing Machinery

T2 - 33rd European Conference on Cognitive Ergonomics

Y2 - 4 October 2022 through 7 October 2022

ER -

Usability testing and trust analysis of a mental health and wellbeing chatbot