Video-based automatic detection of agitated behaviours

Abstract

It is expected that by 2050 153 million people globally, predominantly elderly, will be living with dementia. As the proportion of older people increases, we need to find innovative ways to support their care.

With the easy availability of inexpensive wifi-enabled cameras and the recent development of open-source tools to generate accurate pose-estimation models using video data, automated pose-based activity recognition becomes possible, even for sedentary behaviours. Early research focused on generating features from 3D models built using data from specialised RGB cameras embodying additional depth sensors. However, conventional RGB cameras are an attractive option for data capture due to their low cost and ease of use. We show that 2D models can be as informative as 3D models as a basis for sedentary activity recognition, thus opening the way for the development of affordable and accessible systems to support care at home for people with dementia.

This thesis explores the potential of using skeletal keypoints in body and hand models developed from video data to automatically identify agitation in people living with dementia, assessing the frequency and intensity of agitated episodes. As many older adults are largely sedentary, we focus on agitated behaviours displayed whilst a person is seated within view of a single RGB video camera.

Agitation can manifest itself in several different ways, including repetitive mannerisms such as scratching, picking, or rubbing. Using hand and wrist keypoints from pose estimation models generated from video, we learn to recognise repetitive hand movements using a small number of discriminative poses. Our method, which is the first to use hand keypoints for recognising non-communicative gestures, is invariant to the speed of hand movement, recognising nearly 80% of the types of hand movement. Moreover, when some user-specific information was included in the training data, our model identified over 98% of hand movements. In addition to learning to recognise agitated behaviour, recognising changes in the intensity of agitation can provide important metrics for monitoring changes in behaviour. By first learning characteristics of the movements corresponding to particular agitated behaviours, our model was able to detect over 90% of changes in the intensity of repetitive hand movements.

Although activity recognition based on pose estimation models has attracted much interest, most work has focused on recognising activities from pre-segmented video sequences. However, identifying and assessing agitation requires the detection of (often occasional) abnormal behaviour from within long periods of continuous data. Correspondingly, we also present a method to segment streamed data automatically into sequences containing similar types of hand movements. Our approach improves the detection of agitation by 67% relative to the moving window approach used by other agitation detection models. Additionally, an effective method of cleaning hand keypoint data is proposed.

The findings of this research will support the development of unintrusive systems for monitoring agitated behaviour. Automatic monitoring will enable better understanding of agitated behaviours in dementia, providing accurate and objective metrics for clinicians to determine appropriate care pathways, and enable carers to be alerted when intervention is required.

Date of Award	Sept 2022
Original language	English
Sponsors	Department for the Economy
Supervisor	Shuai Zhang (Supervisor) & Bryan Scotney (Supervisor)