Anomaly detection in performance regression testing by transaction profile estimation

As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload-independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile (TP). The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the 'TP run report' and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques.