LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge

Zumin Wang, Ji-Yu Tian, Hui Fang, Luke Chen, Jing Qin

Research output: Contribution to journalArticlepeer-review

45 Citations (Scopus)
198 Downloads (Pure)

Abstract

Log anomaly detection on edge devices is the key to enhance edge security when deploying IoT systems. Despite the success of many newly proposed deep learning based log anomaly detection methods, handling large-scale logs on edge devices is still a bottleneck due to the limited computational power on these devices to fulfil the real-time processing requirement for accurate anomaly detection. In this work, we propose a novel lightweight log anomaly detection algorithm, named LightLog, to tackle this research gap. In specific, we achieve real-time processing speed on the task via two aspects: (i) creation of a low-dimensional semantic vector space based on word2vec and post-processing algorithms (PPA); and (ii) design of a lightweight temporal convolutional network (TCN) for the detection. These two components significantly reduce the number of parameters and computations of a standard TCN while improving the detection performance. Experimental results show that our LightLog outperforms several benchmarking methods, namely DeepLog, LogAnomaly and RobustLog, by achieving 97.0 F1 score on HDFS Dataset and 97.2 F1 score on BGL with smallest model size. This effective yet efficient method paves the way to the deployment of log anomaly detection on the edge. Our source code and datasets are freely available on https://github.com/Aquariuaa/LightLog.
Original languageEnglish
Article number108616
JournalComputer Networks
Volume203
Early online date10 Nov 2021
DOIs
Publication statusPublished (in print/issue) - 11 Feb 2022

Bibliographical note

Funding Information:
This work was supported by the Youth Fund Project of the National Natural Science Foundation of China under grant 62002038 .

Publisher Copyright:
© 2021 Elsevier B.V.

Keywords

  • Edge computing
  • Global average pooling
  • Log anomaly detection
  • Pointwise-convolution
  • Temporal convolutional network

Fingerprint

Dive into the research topics of 'LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge'. Together they form a unique fingerprint.

Cite this