Calibration of uncertainty in the active learning of machine learning force fields

Adam Thomas-Mitchell; Glenn Ivan Hawe; P. L. A. Popelier

doi:10.1088/2632-2153/ad0ab5

Calibration of uncertainty in the active learning of machine learning force fields

Adam Thomas-Mitchell
, Glenn Ivan Hawe
, P. L. A. Popelier

Research output: Contribution to journal › Article › peer-review

5 Citations (Scopus)

177 Downloads (Pure)

Abstract

FFLUX is a Machine Learning Force Field that uses the Maximum Expected Prediction Error (MEPE) active learning algorithm to improve the efficiency of model training. MEPE uses the predictive uncertainty of a Gaussian Process to balance exploration and exploitation when selecting the next training sample. However, the predictive uncertainty of a Gaussian Process is unlikely to be accurate or precise immediately after training. We hypothesize that calibrating the uncertainty quantification within MEPE will improve active learning performance. We develop and test two methods to improve uncertainty estimates: post-hoc calibration of predictive uncertainty using the CRUDE algorithm, and replacing the Gaussian Process with a Student-t Process. We investigate the impact of these methods on MEPE for single sample and batch sample active learning. Our findings suggest that post-hoc calibration does not improve the performance of active learning using the MEPE method. However, we do find that the Student-t Process can outperform active learning strategies and random sampling using a Gaussian Process if the training set is sufficiently large.

Original language	English
Article number	045034
Pages (from-to)	1-19
Number of pages	19
Journal	Machine Learning: Science and Technology
Volume	4
Issue number	4
Early online date	23 Nov 2023
DOIs	https://doi.org/10.1088/2632-2153/ad0ab5
Publication status	Published online - 23 Nov 2023

Bibliographical note

Publisher Copyright:
© 2023 The Author(s). Published by IOP Publishing Ltd.

Data Availability Statement

The data that support the findings of this study are openly available at the following URL/DOI: https:// github.com/AdamThomas-Mitchell/uqlab/tree/master/uqlab/data.

Funding

We are grateful for the use of the computing resources from the Northern Ireland High Performance Computing (NI-HPC) service. We are also grateful to Yulian Manchev for providing the dataset. P L A P is grateful to the European Research Council (ERC) for the award of an Advanced Grant underwritten by the UKRI-funded Frontier Research grant EP/XO24393/1.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure
SDG 12 Responsible Consumption and Production

Keywords

machine learning force fields
Gaussian process regression (GPR)
calibration
Active learning
uncertainty quantification
active learning
Gaussian process

Access to Document

10.1088/2632-2153/ad0ab5Licence: CC BY

Thomas-Mitchell_2023_Mach._Learn.__Sci._Technol._4_045034Final published version, 3.26 MBLicence: CC BY

Cite this

@article{85c8408dc41b49ee9969a4851fb75b94,

title = "Calibration of uncertainty in the active learning of machine learning force fields",

abstract = "FFLUX is a Machine Learning Force Field that uses the Maximum Expected Prediction Error (MEPE) active learning algorithm to improve the efficiency of model training. MEPE uses the predictive uncertainty of a Gaussian Process to balance exploration and exploitation when selecting the next training sample. However, the predictive uncertainty of a Gaussian Process is unlikely to be accurate or precise immediately after training. We hypothesize that calibrating the uncertainty quantification within MEPE will improve active learning performance. We develop and test two methods to improve uncertainty estimates: post-hoc calibration of predictive uncertainty using the CRUDE algorithm, and replacing the Gaussian Process with a Student-t Process. We investigate the impact of these methods on MEPE for single sample and batch sample active learning. Our findings suggest that post-hoc calibration does not improve the performance of active learning using the MEPE method. However, we do find that the Student-t Process can outperform active learning strategies and random sampling using a Gaussian Process if the training set is sufficiently large.",

keywords = "machine learning force fields, Gaussian process regression (GPR), calibration, Active learning, uncertainty quantification, active learning, Gaussian process",

author = "Adam Thomas-Mitchell and Hawe, \{Glenn Ivan\} and Popelier, \{P. L. A.\}",

note = "Publisher Copyright: {\textcopyright} 2023 The Author(s). Published by IOP Publishing Ltd.",

year = "2023",

month = nov,

day = "23",

doi = "10.1088/2632-2153/ad0ab5",

language = "English",

volume = "4",

pages = "1--19",

journal = "Machine Learning: Science and Technology",

issn = "2632-2153",

publisher = "Institute of Physics Publishing",

number = "4",

}

TY - JOUR

T1 - Calibration of uncertainty in the active learning of machine learning force fields

AU - Thomas-Mitchell, Adam

AU - Hawe, Glenn Ivan

AU - Popelier, P. L. A.

PY - 2023/11/23

Y1 - 2023/11/23

N2 - FFLUX is a Machine Learning Force Field that uses the Maximum Expected Prediction Error (MEPE) active learning algorithm to improve the efficiency of model training. MEPE uses the predictive uncertainty of a Gaussian Process to balance exploration and exploitation when selecting the next training sample. However, the predictive uncertainty of a Gaussian Process is unlikely to be accurate or precise immediately after training. We hypothesize that calibrating the uncertainty quantification within MEPE will improve active learning performance. We develop and test two methods to improve uncertainty estimates: post-hoc calibration of predictive uncertainty using the CRUDE algorithm, and replacing the Gaussian Process with a Student-t Process. We investigate the impact of these methods on MEPE for single sample and batch sample active learning. Our findings suggest that post-hoc calibration does not improve the performance of active learning using the MEPE method. However, we do find that the Student-t Process can outperform active learning strategies and random sampling using a Gaussian Process if the training set is sufficiently large.

AB - FFLUX is a Machine Learning Force Field that uses the Maximum Expected Prediction Error (MEPE) active learning algorithm to improve the efficiency of model training. MEPE uses the predictive uncertainty of a Gaussian Process to balance exploration and exploitation when selecting the next training sample. However, the predictive uncertainty of a Gaussian Process is unlikely to be accurate or precise immediately after training. We hypothesize that calibrating the uncertainty quantification within MEPE will improve active learning performance. We develop and test two methods to improve uncertainty estimates: post-hoc calibration of predictive uncertainty using the CRUDE algorithm, and replacing the Gaussian Process with a Student-t Process. We investigate the impact of these methods on MEPE for single sample and batch sample active learning. Our findings suggest that post-hoc calibration does not improve the performance of active learning using the MEPE method. However, we do find that the Student-t Process can outperform active learning strategies and random sampling using a Gaussian Process if the training set is sufficiently large.

KW - machine learning force fields

KW - Gaussian process regression (GPR)

KW - calibration

KW - Active learning

KW - uncertainty quantification

KW - active learning

KW - Gaussian process

UR - https://doi.org/10.1088/2632-2153/ad0ab5

UR - https://www.scopus.com/pages/publications/85179473462

U2 - 10.1088/2632-2153/ad0ab5

DO - 10.1088/2632-2153/ad0ab5

M3 - Article

SN - 2632-2153

VL - 4

SP - 1

EP - 19

JO - Machine Learning: Science and Technology

JF - Machine Learning: Science and Technology

IS - 4

M1 - 045034

ER -

Calibration of uncertainty in the active learning of machine learning force fields

Abstract

Bibliographical note

Data Availability Statement

Funding

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint