EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments

Haider Raza, Girijesh Prasad, Yuhua Li

Research output: Chapter in Book/Report/Conference proceedingChapter

10 Citations (Scopus)

Abstract

Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. In a time-series data, detecting the dataset shift point, where the distribution changes its properties is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents a novel method to detect the shift-point based on a two-stage structure involving Exponentially WeightedMoving Average (EWMA) chart and Kolmogorov-Smirnov test, which substantially reduces type-I error rate. The algorithm is suitable to be run in real-time. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show effectiveness of the proposed approach in terms of decreased type-I error and tolerable increase in detection time delay.
LanguageEnglish
Title of host publicationArtificial Intelligence Applications and Innovations
EditorsHarris Papadopoulos, Andreas. S. Andreou, Lazaros IIiadis, IIias Maglogiannis
Place of PublicationLondon
Pages625-635
Volume412
DOIs
Publication statusPublished - 1 Oct 2013

Fingerprint

Time series
Time delay
Monitoring
Experiments

Cite this

Raza, H., Prasad, G., & Li, Y. (2013). EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments. In H. Papadopoulos, A. S. Andreou, L. IIiadis, & II. Maglogiannis (Eds.), Artificial Intelligence Applications and Innovations (Vol. 412, pp. 625-635). London. https://doi.org/10.1007/978-3-642-41142-7_63
Raza, Haider ; Prasad, Girijesh ; Li, Yuhua. / EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments. Artificial Intelligence Applications and Innovations. editor / Harris Papadopoulos ; Andreas. S. Andreou ; Lazaros IIiadis ; IIias Maglogiannis. Vol. 412 London, 2013. pp. 625-635
@inbook{014df76803f54e21be7dcc2813adc605,
title = "EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments",
abstract = "Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. In a time-series data, detecting the dataset shift point, where the distribution changes its properties is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents a novel method to detect the shift-point based on a two-stage structure involving Exponentially WeightedMoving Average (EWMA) chart and Kolmogorov-Smirnov test, which substantially reduces type-I error rate. The algorithm is suitable to be run in real-time. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show effectiveness of the proposed approach in terms of decreased type-I error and tolerable increase in detection time delay.",
author = "Haider Raza and Girijesh Prasad and Yuhua Li",
note = "9th IFIP WG 12.5 International Conference, AIAI 2013, Paphos, Cyprus, September 30 – October 2, 2013, Proceedings",
year = "2013",
month = "10",
day = "1",
doi = "10.1007/978-3-642-41142-7_63",
language = "English",
isbn = "978-3-642-41141-0",
volume = "412",
pages = "625--635",
editor = "Harris Papadopoulos and Andreou, {Andreas. S.} and Lazaros IIiadis and IIias Maglogiannis",
booktitle = "Artificial Intelligence Applications and Innovations",

}

Raza, H, Prasad, G & Li, Y 2013, EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments. in H Papadopoulos, AS Andreou, L IIiadis & II Maglogiannis (eds), Artificial Intelligence Applications and Innovations. vol. 412, London, pp. 625-635. https://doi.org/10.1007/978-3-642-41142-7_63

EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments. / Raza, Haider; Prasad, Girijesh; Li, Yuhua.

Artificial Intelligence Applications and Innovations. ed. / Harris Papadopoulos; Andreas. S. Andreou; Lazaros IIiadis; IIias Maglogiannis. Vol. 412 London, 2013. p. 625-635.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments

AU - Raza, Haider

AU - Prasad, Girijesh

AU - Li, Yuhua

N1 - 9th IFIP WG 12.5 International Conference, AIAI 2013, Paphos, Cyprus, September 30 – October 2, 2013, Proceedings

PY - 2013/10/1

Y1 - 2013/10/1

N2 - Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. In a time-series data, detecting the dataset shift point, where the distribution changes its properties is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents a novel method to detect the shift-point based on a two-stage structure involving Exponentially WeightedMoving Average (EWMA) chart and Kolmogorov-Smirnov test, which substantially reduces type-I error rate. The algorithm is suitable to be run in real-time. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show effectiveness of the proposed approach in terms of decreased type-I error and tolerable increase in detection time delay.

AB - Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. In a time-series data, detecting the dataset shift point, where the distribution changes its properties is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents a novel method to detect the shift-point based on a two-stage structure involving Exponentially WeightedMoving Average (EWMA) chart and Kolmogorov-Smirnov test, which substantially reduces type-I error rate. The algorithm is suitable to be run in real-time. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show effectiveness of the proposed approach in terms of decreased type-I error and tolerable increase in detection time delay.

U2 - 10.1007/978-3-642-41142-7_63

DO - 10.1007/978-3-642-41142-7_63

M3 - Chapter

SN - 978-3-642-41141-0

VL - 412

SP - 625

EP - 635

BT - Artificial Intelligence Applications and Innovations

A2 - Papadopoulos, Harris

A2 - Andreou, Andreas. S.

A2 - IIiadis, Lazaros

A2 - Maglogiannis, IIias

CY - London

ER -

Raza H, Prasad G, Li Y. EWMA Based Two-Stage Dataset Shift-Detection in Non-stationary Environments. In Papadopoulos H, Andreou AS, IIiadis L, Maglogiannis II, editors, Artificial Intelligence Applications and Innovations. Vol. 412. London. 2013. p. 625-635 https://doi.org/10.1007/978-3-642-41142-7_63