Dataset Shift Detection in Non-stationary Environments Using EWMA Charts

Haider Raza, G Prasad, Yuhua Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. Detecting the dataset shift point in the time-series data, where the distribution of time-series changes its properties, is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents an algorithm to detect the shift-point in a non-stationary time-series data. The proposed method detects the shift-point based on an exponentially weighted moving average (EWMA) control chart for auto-correlated observations. This algorithm is suitable to be run in real-time and monitors the data to detect the dataset shift. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show that all the dataset-shifts are detected without the delay.
LanguageEnglish
Title of host publicationUnknown Host Publication
Pages3151 -3156
Number of pages6
DOIs
Publication statusPublished - 13 Oct 2013
EventIEEE International Conference on Systems, Man, and Cybernetics - Manchester, UK.
Duration: 13 Oct 2013 → …

Conference

ConferenceIEEE International Conference on Systems, Man, and Cybernetics
Period13/10/13 → …

Fingerprint

Time series
Monitoring
Experiments
Control charts

Cite this

Raza, Haider ; Prasad, G ; Li, Yuhua. / Dataset Shift Detection in Non-stationary Environments Using EWMA Charts. Unknown Host Publication. 2013. pp. 3151 -3156
@inproceedings{70359e13e9524728b42e19294d450bb2,
title = "Dataset Shift Detection in Non-stationary Environments Using EWMA Charts",
abstract = "Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. Detecting the dataset shift point in the time-series data, where the distribution of time-series changes its properties, is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents an algorithm to detect the shift-point in a non-stationary time-series data. The proposed method detects the shift-point based on an exponentially weighted moving average (EWMA) control chart for auto-correlated observations. This algorithm is suitable to be run in real-time and monitors the data to detect the dataset shift. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show that all the dataset-shifts are detected without the delay.",
author = "Haider Raza and G Prasad and Yuhua Li",
year = "2013",
month = "10",
day = "13",
doi = "10.1109/SMC.2013.537",
language = "English",
pages = "3151 --3156",
booktitle = "Unknown Host Publication",

}

Raza, H, Prasad, G & Li, Y 2013, Dataset Shift Detection in Non-stationary Environments Using EWMA Charts. in Unknown Host Publication. pp. 3151 -3156, IEEE International Conference on Systems, Man, and Cybernetics, 13/10/13. https://doi.org/10.1109/SMC.2013.537

Dataset Shift Detection in Non-stationary Environments Using EWMA Charts. / Raza, Haider; Prasad, G; Li, Yuhua.

Unknown Host Publication. 2013. p. 3151 -3156.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Dataset Shift Detection in Non-stationary Environments Using EWMA Charts

AU - Raza, Haider

AU - Prasad, G

AU - Li, Yuhua

PY - 2013/10/13

Y1 - 2013/10/13

N2 - Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. Detecting the dataset shift point in the time-series data, where the distribution of time-series changes its properties, is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents an algorithm to detect the shift-point in a non-stationary time-series data. The proposed method detects the shift-point based on an exponentially weighted moving average (EWMA) control chart for auto-correlated observations. This algorithm is suitable to be run in real-time and monitors the data to detect the dataset shift. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show that all the dataset-shifts are detected without the delay.

AB - Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. Detecting the dataset shift point in the time-series data, where the distribution of time-series changes its properties, is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents an algorithm to detect the shift-point in a non-stationary time-series data. The proposed method detects the shift-point based on an exponentially weighted moving average (EWMA) control chart for auto-correlated observations. This algorithm is suitable to be run in real-time and monitors the data to detect the dataset shift. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show that all the dataset-shifts are detected without the delay.

U2 - 10.1109/SMC.2013.537

DO - 10.1109/SMC.2013.537

M3 - Conference contribution

SP - 3151

EP - 3156

BT - Unknown Host Publication

ER -