How Usable Are Usability Tests?: Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

The potential for technology to support memory or enhance cognition for people living with dementia is currently an area of significant interest. The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia. This paper reports on a study that examines the value of several common usability testing protocols, methods and metrics when used to evaluate the usability of a new personalised reminiscence ‘app’ called ‘InspireD’. InspireD is a bespoke app designed to support personalised reminiscence for people living with dementia. The app was co-created and designed with input from people living with dementia and their caregivers. The research question was ‘are standard usability protocols appropriate to assess the usability of apps for individuals living with dementia?’ The study aimed to determine on determining the appropriateness of methods commonly used to evaluate usability: concurrent think-aloud protocol; video recording and audio recording; metrics such as task completion rates; task completion times; single ease questions before and after task completion; and the systematic usability scale scores. Seven couples comprising a person with dementia with their primary caregiver evaluated the perceived usefulness and the level of user acceptance of the InspireD app. The study indicated that observation and recording of task completion rates and times produced the most reliable results, while the think-aloud methodology was difficult for the people living with dementia and did not produce any reliable data. Thinking-aloud whilst doing a task was perhaps a distraction since it requires a higher cognitive load. The systematic usability scale score which is derived from a post-test instrument is not reliable, as it had no association with the task completion times.
LanguageEnglish
Title of host publicationNew Technologies to Improve Patient Rehabilitation
Subtitle of host publication4th Workshop, REHAB 2016, Revised Selected Papers
EditorsHabib M Fardoun, Ahlam A M Hassan, M Elena de la Guía
Chapter10
Pages126-143
Number of pages18
ISBN (Electronic)978-3-030-16785-1
DOIs
Publication statusPublished - 28 Mar 2019
EventREHAB 2016: 4th Workshop on ICTs for Improving Patients' Rehabilitation Research Techniques - Universidade Lusófona de Humanidades e Tecnologias, Lisbon, Portugal
Duration: 13 Oct 201614 Oct 2016
http://www.rehab-workshop.org/2016/

Publication series

NameCommunications in Computer and Information Science
Volume1002
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937

Workshop

WorkshopREHAB 2016
Abbreviated titleREHAB
CountryPortugal
CityLisbon
Period13/10/1614/10/16
Internet address

Fingerprint

Application programs
Testing
Network protocols
Audio recordings
Video recording
Data storage equipment

Keywords

  • Usability protocols
  • Usability
  • User experience
  • UX
  • Human Computer Interaction (HCI)
  • User studies
  • Evaluation
  • Reminiscence
  • Apps
  • Assistive Technology
  • Human computer interaction
  • Assistive technology

Cite this

Gibson, A., McCauley, C-O., Mulvenna, M., Ryan, A., Laird, L., Curran, K., ... Bond, RR. (2019). How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia. In H. M. Fardoun, A. A. M. Hassan, & M. E. de la Guía (Eds.), New Technologies to Improve Patient Rehabilitation: 4th Workshop, REHAB 2016, Revised Selected Papers (pp. 126-143). (Communications in Computer and Information Science; Vol. 1002). https://doi.org/10.1007/978-3-030-16785-1_10
Gibson, Aideen ; McCauley, Claire-Odile ; Mulvenna, Maurice ; Ryan, A ; Laird, Liz ; Curran, Kevin ; Bunting, B ; Ferry, Finola R ; Bond, RR. / How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia. New Technologies to Improve Patient Rehabilitation: 4th Workshop, REHAB 2016, Revised Selected Papers. editor / Habib M Fardoun ; Ahlam A M Hassan ; M Elena de la Guía. 2019. pp. 126-143 (Communications in Computer and Information Science).
@inbook{41714e95210d40a591755cd0ecdff345,
title = "How Usable Are Usability Tests?: Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia",
abstract = "The potential for technology to support memory or enhance cognition for people living with dementia is currently an area of significant interest. The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia. This paper reports on a study that examines the value of several common usability testing protocols, methods and metrics when used to evaluate the usability of a new personalised reminiscence ‘app’ called ‘InspireD’. InspireD is a bespoke app designed to support personalised reminiscence for people living with dementia. The app was co-created and designed with input from people living with dementia and their caregivers. The research question was ‘are standard usability protocols appropriate to assess the usability of apps for individuals living with dementia?’ The study aimed to determine on determining the appropriateness of methods commonly used to evaluate usability: concurrent think-aloud protocol; video recording and audio recording; metrics such as task completion rates; task completion times; single ease questions before and after task completion; and the systematic usability scale scores. Seven couples comprising a person with dementia with their primary caregiver evaluated the perceived usefulness and the level of user acceptance of the InspireD app. The study indicated that observation and recording of task completion rates and times produced the most reliable results, while the think-aloud methodology was difficult for the people living with dementia and did not produce any reliable data. Thinking-aloud whilst doing a task was perhaps a distraction since it requires a higher cognitive load. The systematic usability scale score which is derived from a post-test instrument is not reliable, as it had no association with the task completion times.",
keywords = "Usability protocols, Usability, User experience, UX, Human Computer Interaction (HCI), User studies, Evaluation, Reminiscence, Apps, Assistive Technology, Human computer interaction, Assistive technology",
author = "Aideen Gibson and Claire-Odile McCauley and Maurice Mulvenna and A Ryan and Liz Laird and Kevin Curran and B Bunting and Ferry, {Finola R} and RR Bond",
year = "2019",
month = "3",
day = "28",
doi = "10.1007/978-3-030-16785-1_10",
language = "English",
isbn = "978-3-030-16784-4",
series = "Communications in Computer and Information Science",
pages = "126--143",
editor = "Fardoun, {Habib M} and Hassan, {Ahlam A M } and {de la Gu{\'i}a}, {M Elena }",
booktitle = "New Technologies to Improve Patient Rehabilitation",

}

Gibson, A, McCauley, C-O, Mulvenna, M, Ryan, A, Laird, L, Curran, K, Bunting, B, Ferry, FR & Bond, RR 2019, How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia. in HM Fardoun, AAM Hassan & ME de la Guía (eds), New Technologies to Improve Patient Rehabilitation: 4th Workshop, REHAB 2016, Revised Selected Papers. Communications in Computer and Information Science, vol. 1002, pp. 126-143, REHAB 2016, Lisbon, Portugal, 13/10/16. https://doi.org/10.1007/978-3-030-16785-1_10

How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia. / Gibson, Aideen; McCauley, Claire-Odile; Mulvenna, Maurice; Ryan, A; Laird, Liz; Curran, Kevin; Bunting, B; Ferry, Finola R; Bond, RR.

New Technologies to Improve Patient Rehabilitation: 4th Workshop, REHAB 2016, Revised Selected Papers. ed. / Habib M Fardoun; Ahlam A M Hassan; M Elena de la Guía. 2019. p. 126-143 (Communications in Computer and Information Science; Vol. 1002).

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - How Usable Are Usability Tests?

T2 - Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia

AU - Gibson, Aideen

AU - McCauley, Claire-Odile

AU - Mulvenna, Maurice

AU - Ryan, A

AU - Laird, Liz

AU - Curran, Kevin

AU - Bunting, B

AU - Ferry, Finola R

AU - Bond, RR

PY - 2019/3/28

Y1 - 2019/3/28

N2 - The potential for technology to support memory or enhance cognition for people living with dementia is currently an area of significant interest. The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia. This paper reports on a study that examines the value of several common usability testing protocols, methods and metrics when used to evaluate the usability of a new personalised reminiscence ‘app’ called ‘InspireD’. InspireD is a bespoke app designed to support personalised reminiscence for people living with dementia. The app was co-created and designed with input from people living with dementia and their caregivers. The research question was ‘are standard usability protocols appropriate to assess the usability of apps for individuals living with dementia?’ The study aimed to determine on determining the appropriateness of methods commonly used to evaluate usability: concurrent think-aloud protocol; video recording and audio recording; metrics such as task completion rates; task completion times; single ease questions before and after task completion; and the systematic usability scale scores. Seven couples comprising a person with dementia with their primary caregiver evaluated the perceived usefulness and the level of user acceptance of the InspireD app. The study indicated that observation and recording of task completion rates and times produced the most reliable results, while the think-aloud methodology was difficult for the people living with dementia and did not produce any reliable data. Thinking-aloud whilst doing a task was perhaps a distraction since it requires a higher cognitive load. The systematic usability scale score which is derived from a post-test instrument is not reliable, as it had no association with the task completion times.

AB - The potential for technology to support memory or enhance cognition for people living with dementia is currently an area of significant interest. The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia. This paper reports on a study that examines the value of several common usability testing protocols, methods and metrics when used to evaluate the usability of a new personalised reminiscence ‘app’ called ‘InspireD’. InspireD is a bespoke app designed to support personalised reminiscence for people living with dementia. The app was co-created and designed with input from people living with dementia and their caregivers. The research question was ‘are standard usability protocols appropriate to assess the usability of apps for individuals living with dementia?’ The study aimed to determine on determining the appropriateness of methods commonly used to evaluate usability: concurrent think-aloud protocol; video recording and audio recording; metrics such as task completion rates; task completion times; single ease questions before and after task completion; and the systematic usability scale scores. Seven couples comprising a person with dementia with their primary caregiver evaluated the perceived usefulness and the level of user acceptance of the InspireD app. The study indicated that observation and recording of task completion rates and times produced the most reliable results, while the think-aloud methodology was difficult for the people living with dementia and did not produce any reliable data. Thinking-aloud whilst doing a task was perhaps a distraction since it requires a higher cognitive load. The systematic usability scale score which is derived from a post-test instrument is not reliable, as it had no association with the task completion times.

KW - Usability protocols

KW - Usability

KW - User experience

KW - UX

KW - Human Computer Interaction (HCI)

KW - User studies

KW - Evaluation

KW - Reminiscence

KW - Apps

KW - Assistive Technology

KW - Human computer interaction

KW - Assistive technology

UR - http://www.scopus.com/inward/record.url?scp=85064559458&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-16785-1_10

DO - 10.1007/978-3-030-16785-1_10

M3 - Chapter

SN - 978-3-030-16784-4

T3 - Communications in Computer and Information Science

SP - 126

EP - 143

BT - New Technologies to Improve Patient Rehabilitation

A2 - Fardoun, Habib M

A2 - Hassan, Ahlam A M

A2 - de la Guía, M Elena

ER -

Gibson A, McCauley C-O, Mulvenna M, Ryan A, Laird L, Curran K et al. How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia. In Fardoun HM, Hassan AAM, de la Guía ME, editors, New Technologies to Improve Patient Rehabilitation: 4th Workshop, REHAB 2016, Revised Selected Papers. 2019. p. 126-143. (Communications in Computer and Information Science). https://doi.org/10.1007/978-3-030-16785-1_10