Anomaly detection in performance regression testing by transaction profile estimation

Shadi Ghaith, Miao Wang, Philip Perry, Zhen Ming Jiang, Pat O'Sullivan, John Murphy

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload-independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile (TP). The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the 'TP run report' and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques.

Original languageEnglish
Pages (from-to)4-39
Number of pages36
JournalSoftware Testing Verification and Reliability
Volume26
Issue number1
Early online date9 Mar 2015
DOIs
Publication statusPublished - 1 Jan 2016

    Fingerprint

Keywords

  • performance models
  • performance regression testing
  • software update

Cite this