Multiple Rewards for Unsupervised Q-Learning and HBOS Anomaly Detection

Applying multiple rewards of equal value to QLearning is necessary for accurate anomaly detection, but there is so far little research done into the topic. In this paper, the authors make use of the HBOS algorithm to establish multiple bandpass ranges, which are then used in the training of a Q-Learning algorithm, which is itself based on the Gridworld game, to create Multiple Rewards for Q Learning Version 2 (MRQLv2). A brief introduction into the concepts and formulae behind Q-Learning and the HBOS algorithm are included. The methodology section shows how the bounds of the bandpass filters are created; how the data is prepared; how the algorithm handles different types of data; the creation of category maps and the use of parameters. The full MRQLv2 algorithm is tested on the Statlog Shuttle and Satellite UCI datasets. The results show a recall of 1, a precision of 0.946408 and a ROC-AUC score of 0.973204 on the Shuttle dataset. These results are compared against a variety of machine learning, deep learning and hybrid models including; OCSVM, MF, kNN and Fast kNN, GNNs, RNNs, MCOD and Approx-Storm using precision, recall and averaged AUC scores. The results show a significant improvement on previous models for the Shuttle dataset, but more research needs to be done to improve the performance of the MRQLv2 algorithm
overall.