Assessing the Effect of Data Complexity and Instance Overlap Issues on Imbalanced Learning

Most machine learning (ML) algorithms work best when the samples in each class are almost equal. However, if a dataset has imbalanced samples then the ML model can achieve a high accuracy by just predicting the majority of classes and achieving a high classifier performance. Different resampling methods are used to handle this imbalance issue by creating synthetic data in the minority class or removing samples from the majority class until the classes are balanced. However, using these resampling methods can harm the dataset performance by creating synthetic data samples in a dataset that already suffers from overlap and other data intrinsic issues. Recently, dataset complexity measures, by observing the characteristics of a dataset, are often used in machine learning tasks as intrinsic descriptors to calculate the difficulty of a classification problem. This study investigates how resampling methods can affect the dataset in terms of complexity, overlapping, and classification accuracy. This is achieved by monitoring 22 different measurements related to data complexity measures obtained from 20 public and pre-processed datasets from the KEEL repository. Our empirical findings demonstrate a strong positive correlation between the resampling methods and the dataset's complexity and classification accuracy. We also posit that the main reason for poor classification accuracy on an imbalanced dataset is not due to its imbalanced nature but due to the other intrinsic characteristics of the dataset. Finally, we advocate that pre-processing methods and ML algorithms should be based on the dataset's specific properties, rather than being chosen on an ad hoc basis.