Model Variance in Neural Network Models for Forecasting Chaotic Systems

Kevin Nolan; Mark McCartney; David H. Glass; Samuel Moore

doi:10.1007/978-3-031-78857-4

Model Variance in Neural Network Models for Forecasting Chaotic Systems

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

Abstract

While the non-linearity and sensitivity of chaotic systems make the forecasting of their future behaviour challenging, the ability of machine learning techniques to accurately represent complex dynamics has placed them at the forefront of this field. However, small differences in the model used can lead to noticeable differences in observable model performance due to the systems’ sensitivity. This can result in even seemingly inconsequential changes to the training process causing practical differences in model performance. In this study performance metrics for both next-step prediction accuracy and a novel metric for the duration of accurate prediction have been calculated for a number of long-short term memory (LSTM) models, and their variance studied. The results highlight the causes and quantifies the scale of inconsistency in model performance that can be found during seemingly equivalent training scenarios, and concludes that such variance in model performance is non-trivial.

Original language	English
Title of host publication	Advances in Computational Intelligence Systems
Subtitle of host publication	Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK
Editors	Huiru Zheng, David Glass, Maurice Mulvenna, Jun Liu, Hui Wang
Publisher	Springer Cham
Pages	119-125
Number of pages	7
Volume	1462
Edition	1
ISBN (Electronic)	978-3-031-78857-4
ISBN (Print)	978-3-031-78856-7
DOIs	https://doi.org/10.1007/978-3-031-78857-4
Publication status	Published online - 8 Jan 2025
Event	23rd Annual UK Workshop on Computational Intelligence 2024 - Ulster University, Belfast, Belfast, Northern Ireland Duration: 2 Sept 2024 → 4 Sept 2024 https://computing.ulster.ac.uk/ZhengLab/UKCI2024/

Workshop

Workshop	23rd Annual UK Workshop on Computational Intelligence 2024
Abbreviated title	UKCI 2024
Country/Territory	Northern Ireland
City	Belfast
Period	2/09/24 → 4/09/24
Internet address	https://computing.ulster.ac.uk/ZhengLab/UKCI2024/

Funding

This research is supported by the ARC (Advanced Research Engineering Centre) project. PWC ∗(PricewaterhouseCoopers LLP a limited liability partnership incorporated in England with its registered office office at 1 Embankment Place, London WC2N 6RH) is in receipt of Grant for R&D support from Invest NI for ARC. This project is part-financed by the European Regional Development Fund under the Investment for Growth and Jobs Programme 2014–2020. This work was supported by funding from the Department for the Economy, Northern Ireland.

Keywords

Machine Learning (ML)
Neural Network (NN)
Recurrent Neural Network
Long Short-Term Memory (LSTM)
Chaos
Time Series
Forecasting

Access to Document

10.1007/978-3-031-78857-4

Cite this

Nolan, K., McCartney, M., Glass, D. H., & Moore, S. (2025). Model Variance in Neural Network Models for Forecasting Chaotic Systems. In H. Zheng, D. Glass, M. Mulvenna, J. Liu, & H. Wang (Eds.), Advances in Computational Intelligence Systems: Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK (1 ed., Vol. 1462, pp. 119-125). Springer Cham. Advance online publication. https://doi.org/10.1007/978-3-031-78857-4

Nolan, Kevin ; McCartney, Mark ; Glass, David H. et al. / Model Variance in Neural Network Models for Forecasting Chaotic Systems. Advances in Computational Intelligence Systems: Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK. editor / Huiru Zheng ; David Glass ; Maurice Mulvenna ; Jun Liu ; Hui Wang. Vol. 1462 1. ed. Springer Cham, 2025. pp. 119-125

@inbook{5262c67f0788474fb3b0d9b577e44d42,

title = "Model Variance in Neural Network Models for Forecasting Chaotic Systems",

abstract = "While the non-linearity and sensitivity of chaotic systems make the forecasting of their future behaviour challenging, the ability of machine learning techniques to accurately represent complex dynamics has placed them at the forefront of this field. However, small differences in the model used can lead to noticeable differences in observable model performance due to the systems{\textquoteright} sensitivity. This can result in even seemingly inconsequential changes to the training process causing practical differences in model performance. In this study performance metrics for both next-step prediction accuracy and a novel metric for the duration of accurate prediction have been calculated for a number of long-short term memory (LSTM) models, and their variance studied. The results highlight the causes and quantifies the scale of inconsistency in model performance that can be found during seemingly equivalent training scenarios, and concludes that such variance in model performance is non-trivial. ",

keywords = "Machine Learning (ML), Neural Network (NN), Recurrent Neural Network, Long Short-Term Memory (LSTM), Chaos, Time Series, Forecasting",

author = "Kevin Nolan and Mark McCartney and Glass, \{David H.\} and Samuel Moore",

year = "2025",

month = jan,

day = "8",

doi = "10.1007/978-3-031-78857-4",

language = "English",

isbn = "978-3-031-78856-7",

volume = "1462",

pages = "119--125",

editor = "Huiru Zheng and Glass, \{David \} and Mulvenna, \{Maurice \} and Jun Liu and Hui Wang",

booktitle = "Advances in Computational Intelligence Systems",

publisher = "Springer Cham",

edition = "1",

note = "23rd Annual UK Workshop on Computational Intelligence 2024, UKCI 2024 ; Conference date: 02-09-2024 Through 04-09-2024",

url = "https://computing.ulster.ac.uk/ZhengLab/UKCI2024/",

}

Nolan, K , McCartney, M , Glass, DH & Moore, S 2025, Model Variance in Neural Network Models for Forecasting Chaotic Systems. in H Zheng, D Glass, M Mulvenna, J Liu & H Wang (eds), Advances in Computational Intelligence Systems: Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK. 1 edn, vol. 1462, Springer Cham, pp. 119-125, 23rd Annual UK Workshop on Computational Intelligence 2024, Belfast, Northern Ireland, 2/09/24. https://doi.org/10.1007/978-3-031-78857-4

Model Variance in Neural Network Models for Forecasting Chaotic Systems. / Nolan, Kevin ; McCartney, Mark ; Glass, David H. et al.
Advances in Computational Intelligence Systems: Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK. ed. / Huiru Zheng; David Glass; Maurice Mulvenna; Jun Liu; Hui Wang. Vol. 1462 1. ed. Springer Cham, 2025. p. 119-125.

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

TY - CHAP

T1 - Model Variance in Neural Network Models for Forecasting Chaotic Systems

AU - Nolan, Kevin

AU - McCartney, Mark

AU - Glass, David H.

AU - Moore, Samuel

PY - 2025/1/8

Y1 - 2025/1/8

N2 - While the non-linearity and sensitivity of chaotic systems make the forecasting of their future behaviour challenging, the ability of machine learning techniques to accurately represent complex dynamics has placed them at the forefront of this field. However, small differences in the model used can lead to noticeable differences in observable model performance due to the systems’ sensitivity. This can result in even seemingly inconsequential changes to the training process causing practical differences in model performance. In this study performance metrics for both next-step prediction accuracy and a novel metric for the duration of accurate prediction have been calculated for a number of long-short term memory (LSTM) models, and their variance studied. The results highlight the causes and quantifies the scale of inconsistency in model performance that can be found during seemingly equivalent training scenarios, and concludes that such variance in model performance is non-trivial.

AB - While the non-linearity and sensitivity of chaotic systems make the forecasting of their future behaviour challenging, the ability of machine learning techniques to accurately represent complex dynamics has placed them at the forefront of this field. However, small differences in the model used can lead to noticeable differences in observable model performance due to the systems’ sensitivity. This can result in even seemingly inconsequential changes to the training process causing practical differences in model performance. In this study performance metrics for both next-step prediction accuracy and a novel metric for the duration of accurate prediction have been calculated for a number of long-short term memory (LSTM) models, and their variance studied. The results highlight the causes and quantifies the scale of inconsistency in model performance that can be found during seemingly equivalent training scenarios, and concludes that such variance in model performance is non-trivial.

KW - Machine Learning (ML)

KW - Neural Network (NN)

KW - Recurrent Neural Network

KW - Long Short-Term Memory (LSTM)

KW - Chaos

KW - Time Series

KW - Forecasting

UR - https://pure.ulster.ac.uk/en/publications/5262c67f-0788-474f-b3b0-d9b577e44d42

U2 - 10.1007/978-3-031-78857-4

DO - 10.1007/978-3-031-78857-4

M3 - Chapter

SN - 978-3-031-78856-7

VL - 1462

SP - 119

EP - 125

BT - Advances in Computational Intelligence Systems

A2 - Zheng, Huiru

A2 - Glass, David

A2 - Mulvenna, Maurice

A2 - Liu, Jun

A2 - Wang, Hui

PB - Springer Cham

T2 - 23rd Annual UK Workshop on Computational Intelligence 2024

Y2 - 2 September 2024 through 4 September 2024

ER -

Nolan K , McCartney M , Glass DH , Moore S. Model Variance in Neural Network Models for Forecasting Chaotic Systems. In Zheng H, Glass D, Mulvenna M, Liu J, Wang H, editors, Advances in Computational Intelligence Systems: Contributions Presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK. 1 ed. Vol. 1462. Springer Cham. 2025. p. 119-125 Epub 2025 Jan 8. doi: 10.1007/978-3-031-78857-4

Model Variance in Neural Network Models for Forecasting Chaotic Systems

Abstract

Workshop

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Improving reliability in the internet of things through anomaly detection

Cite this

Model Variance in Neural Network Models for Forecasting Chaotic Systems

Abstract

Workshop

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Student theses

Improving reliability in the internet of things through anomaly detection

Cite this