An Empirical Analysis of State-of-Art Classification Models in an IT Incident Severity Prediction Framework

Salman Ahmed, Muskaan Singh, Brendan Doherty, Effirul Ikhwan Ramlan, Kathryn Harkin, Magda Bucholc, Damien Coyle

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
58 Downloads (Pure)


Large-scale companies across various sectors maintain substantial IT infrastructure to
support their operations and provide quality services for their customers and employees. These IT
operations are managed by teams who deal directly with incident reports (i.e., those generated automatically
through autonomous systems or human operators). (1) Background: Early identification of
major incidents can provide a significant advantage for reducing the disruption to normal business
operations, especially for preventing catastrophic disruptions, such as a complete system shutdown.
(2) Methods: This study conducted an empirical analysis of eleven (11) state-of-the-art models to
predict the severity of these incidents using an industry-led use-case composed of 500,000 records
collected over one year. (3) Results: The datasets were generated from three stakeholders (i.e., agency,
customer, and employee). Separately, the bidirectional encoder representations from transformers
(BERT), the robustly optimized BERT pre-training approach (RoBERTa), the enhanced representation
through knowledge integration (ERNIE 2.0), and the extreme gradient boosting (XGBoost) methods
performed the best for the agency records (93% AUC), while the convolutional neural network
(CNN) was the best model for the rest (employee records at 95% AUC and customer records at
74% AUC, respectively). The average prediction horizon was approximately 150 min, which was
significant for real-time deployment. (4) Conclusions: The study provided a comprehensive analysis
that supported the deployment of artificial intelligence for IT operations (AIOps), specifically for
incident management within large-scale organizations.
Original languageEnglish
Article number3843
Pages (from-to)1-27
Number of pages27
JournalApplied Sciences
Issue number6
Early online date17 Mar 2023
Publication statusPublished (in print/issue) - 17 Mar 2023

Bibliographical note

Funding Information:
We are grateful for access to the tier 2 high-performance computing resources provided by the Northern Ireland High-Performance Computing (NI-HPC) facility, funded by the U.K. Engineering and Physical Sciences Research Council (EPSRC), Grant Nos. EP/T022175/ and EP/W03204X/1. Damien Coyle is supported by the UKRI Turing AI Fellowship 2021–2025 funded by the EPSRC (grant number EP/V025724/1). Salman Ahmed is supported by a George Moore Ph.D. scholarship.

Funding Information:
This work was supported by U.K. Research and Innovation Turing AI Fellowship 2021–2025 funded by the Engineering and Physical Sciences Research Council (grant number EP/V025724/1).

Publisher Copyright:
© 2023 by the authors.


  • IT incidents
  • risk prediction
  • dataset imbalance
  • IT service management (ITSM)
  • Information Technology Infrastructure Library (ITIL)
  • Artificial Intelligence for IT Operations (AIOPS)
  • artificial intelligence for IT operations (AIOps)


Dive into the research topics of 'An Empirical Analysis of State-of-Art Classification Models in an IT Incident Severity Prediction Framework'. Together they form a unique fingerprint.

Cite this