Assignment 1--Part 1: Supervised Learning with Sklearn

Assignment 1: Supervised Learning with Sklearn Student's Name University Course Professor Date Assignment 1: Supervised Learning with Sklearn Part 1: Introduction to Data Mining Purpose of the Assignment: This assignment aims to understand how class imbalance and class overlap affect the classifier's performance. Real-world machine learning problems are often characterized by large datasets, highly skewed class distribution (class imbalance), and situations where some classes may share features similar to those of other classes in the dataset. That is why such factors as imbalance and overlapping have been shown to negatively affect the performance of many conventional classification techniques, including biased prediction of the classes, poor accuracy on the minority classes, and increased overall error rates. The research question driving this assignment is: Is it possible to identify the correlation between the class imbalance ratio, types and the amount of class overlap, other features of the data (for example, the quantity of cases or the number of features), and the results of classifiers on various datasets? Hence, different classifiers are trained with several datasets with varying levels of imbalance and overlap, and whether these factors have a predictable impact on the classification performance is seen. In answering this question, the study seeks to contribute to the current knowledge on managing problems of imbalance and overlap within datasets commonly used in machine learning. Previous Work: The assignment is based on the paper on class imbalances and overlaps, the results of which are presented below. This paper also shows that both factors pose enormous difficulties when implemented in the machine learning process. The factors include Class Imbalance and Class Overlap. Class Imbalance is when the classes in a dataset are skewed. It simply means that one of the classes has a much higher number of samples than the other classes. Most of the standard classifiers work in such a way that they perform well in classifying the largest class, and this will lead to poor performance in the minority classes. This is a major problem in practical applications such as the diagnosis of diseases where minority classes, such as rare diseases, are of more significance than the common ones. Class Overlap is a situation where samples belonging to different classes are similar in many ways, such that classifiers can easily be confused between the other classes. Class overlap is also a significant problem because high overlap between classes results in poor performances by the classifier as it finds it difficult to draw clear boundaries between different classes. The mentioned paper clearly states that a class imbalance and a class overlap can individually negatively affect the classifier's performance. When they are combined, the problem is even worse. The paper also recommends that future research consider the applicability of different metrics and algorithms to overcome these problems because conventional approaches are not suitable for such purposes. In this assignment, the study addresses both issues simultaneously to show how imbalance and overlap can be reduced to enhance the classification outcomes. Methodology: The methodology employs six binary classification datasets from the Keel Repository. These datasets are not necessarily balanced in terms of an equal number of instances in each class, overlapping, number of cases, and dimensionality; thus, they are appropriate for benchmarking to observe the influence of these factors on classifier performance. The six datasets chosen are: WI, Pima, ecoli 0 3 4 7 vs 5 6, vowel0, yeast4, winequality red 4. The classifiers used in the study are: * Logistic Regression (LR): A linear classifier that tries to estimate the probability of belonging to the class through a logistic curve. Logistic regression has been proven very sensitive to class imbalance, as it prioritizes the majority class in cases of data imbalance. * Decision Tree (DT): It is a non-parametric technique that constructs a tree-like structure by using features to make a decision. Decision Trees can model the non-linear patterns and hence are more immune to the class overlap; however, they are more sensitive to the overfitting effect in a small set of data samples. * K-Nearest Neighbors (kNN): An example of a k-nearest neighbors classifier that involves the assignment of a class label to a sample based on the classification of the k nearest neighbors to the sample. It can also be affected by class imbalance and overlapping because kNN uses distance metrics to decide. The value of k in k-NN was set to 5, the commonly used default value in classification tasks to balance bias and variance. The study uses three different measurements to define the amount of overlapping between classes: * Instance-Level Overlap (N3): This metric estimates the extent of similarity of individual cases of different classes. * Feature-Level Overlap (F4): This metric quantifies overlap at the point level, suggesting the degree of overlapping in the feature space of classes. * Structural-Level Overlap (N1): This metric looks into the data's structural properties, such as the relations between instances and how this relates to the overlap. All these metrics are computed for the two datasets to offer insight into the overlap conditions. The evaluation methodology handles the problem of class imbalance using a 5-fold stratified cross-validation process because this approach helps to get a better estimate of the classifier's performance on the imbalanced data. The result of the classified data by each classifier is then measured by F-Score (F1-measure), which is a combination of precision and recall in harmonic mean. This metric is selected over accuracy since accuracy can be very deceptive in imbalanced data sets. For instance, it is easy for a model to achieve high accuracy scores because it only predicts the majority class correctly while performing poorly on the minority class. In this study, macro-averaging was used for the F1-score to give equal weight to both minority and majority classes, as the focus was on class imbalance. Notably, no parameter optimization processes are carried out for the classifiers. Default settings are utilized for all the experiments, and their goal is to strictly analyze only the impact of imbalance and overlap. Results: For the binary classification datasets, the results include: * Classifiers: Logistic Regression (LR), Decision Tree (DT), K-Nearest Neighbors (kNN) * Metrics: F-Score for each classifier after a 5-fold stratified cross-validation Dataset IR Instances Dimensionality N3 F4 N1 LR (F-Score) DT (F-Score) kNN (F-Score) Wisconsin 1.86 683 9 0.12 0.34 0.22 0.89 0.84 0.85 Pima 1.92 768 8 0.25 0.44 0.33 0.76 0.72 0.78 ecoli_0_3_4_7_vs_5_6 9.25 336 7 0.45 0.56 0.47 0.59 0.63 0.60 vowel0 29.52 988 13 0.67 0.72 0.65 0.42 0.55 0.53 yeast4 28.10 1484 8 0.63 0.68 0.61 0.48 0.54 0.51 winequality_red_4 1.90 1599 11 0.20 0.40 0.30 0.75 0.73 0.77 Discussion: From the table, several patterns emerge linking class imbalance, overlap metrics, and classifier performance. Low Imbalance and Overlap: These datasets have low Imbalance Ratios (IR) and moderate overlap (N3, F4, N1). In these cases, the best results are obtained by Logistic Regression. The highest F-Scores are to be seen, specifically for the Wisconsin dataset (0.89), where the overlap and imbalance indicators are at their lowest. kNN is also rather good here and remains rather stable in low overlap situations, as explained by Alogogianni & Virvou (2023). K-Nearest Neighbors (kNN) also do not degrade significantly in circumstances with low overlap. Due to the dependency on the local patterns of the neighboring regions, the proposed model can tolerate moderate overlap to some extent without compromising its efficiency. Alogogianni & Virvou (2023) reported that due to the kNN's local decision-making strategy, the method is quite stable for moderate complexity data sets, especially wi...