A Preliminary Study of Unsupervised Classifiers in a Supervised Environment Compared to Traditional Supervised Classifiers

Fall 2024: CSC-480/680 Project Proposal: A Preliminary Study of Unsupervised Classifiers in a Supervised Environment Compared to Traditional Supervised Classifiers Yalda Rawan American University Introduction to Data Mining Prof. Nathalie Japkowicz Nov / 7 / 2024 Introduction Machine learning encompasses two primary categories: both supervised and unsupervised learning. The datasets have been labeled with supervised learning models, which helps them learn and predict accurately and in a targeted manner. It is highly effective, but acquiring labeled data is costly and time-consuming, which makes supervised learning inappropriate for data-scarce situations. Unlike the above, unsupervised learning is a type that looks at unlabeled data to discover patterns and structures. These models work great for clustering and anomaly detection. Recently, there have been advances that adapt unsupervised k-means and Gaussian Mixture Models (GMM) to be supervised by associating clusters to specific class labels. This research examines whether such adapted unsupervised classifiers can surpass traditional supervised classifiers such as Decision Trees, Support Vector Machines (SVMs), and Neural Networks in performance. This study attempts to cut costs associated with using labeled data by evaluating unsupervised models' fitness and adaptability in supervised scenarios. Background and Motivation Labeled data is an expensive and laborious data source for machine learning models. Supervised classifiers are highly accurate but rely upon the availability of this labeled data, thereby limiting their applicability where data labeling is impossible. Some unsupervised learning methods can be cost-effective ways of generating patterns from unlabeled data, avoiding the expense of labeled data. This study aims to close the gap between the freedom of unsupervised learning and the accuracy of supervised classification. The research will show how unsupervised clustering algorithms, including k-means and Gaussian Mixture Models (GMM), can be adapted for classification tasks not dependent on labeled data. These could be particularly useful when supervised methods are difficult. The results presented in this work open new critical lines of investigation into a paucity of scalable, efficient machine-learning solutions in constrained environments. Problem Definition The overall purpose of this study is to ascertain how unsupervised classifiers that are modified to operate in a supervised manner can perform as effectively as other conventional supervised classifiers. * Extend k-means and Gaussian mixture models (GMM) to enable them to be used well in classification problems. * Report on the findings of these adapted models with those of more conventional supervised classifiers such as decision trees, SVM, and neural networks. * Understand situations where unsupervised models should be applied and be aware of circumstances under which their use is impossible. * Specifically, the project will: Ї Adapt k-means and GMM for use in classification tasks. Ї Compare the performance of these adapted models to traditional supervised classifiers such as decision trees, SVM, and neural networks. Ї Identify scenarios in which unsupervised models are effective and recognize conditions that may not be practical. Significance of the Study This research can be helpful in fields where labeled data is scarce to eliminate the dependency on labeled data. The conclusions may benefit industries and research fields where it is unprofitable to employ labeling for data or labeled data needs to be included, for example, for health care or environmental purposes. Objectives 1 Adapt and Implement: Extend two variants of unsupervised models, k-means and GMM, for supervised classification by assigning proper labels to clusters. 2 Benchmark Performance: The above models are now compared to other models of another type of classification, that is, the traditional supervised methods, by using the standard measures of performance, which are accuracy, precision, Recall, F1-score, and ROC-AUC. 3 Statistical Analysis: Perform hypothesis tests to check the level of the performance differences. 4 Provide Insights: Explain what data characteristics and conditions make it worthwhile or impractical to apply unsupervised models. Literature Review Supervised and Unsupervised Learning Machine learning is primarily divided into two categories: supervised and unsupervised learning. Supervised learning requires labeled data, where input corresponds to known outputs. It is highly effective in tasks like image classification, spam detection, and price forecasting (Tishan, 2023). However, the cost and effort of labeling data can significantly limit this dependency on labeled data. On the contrary, unsupervised learning is based on performing structure discovery in an unlabeled dataset. Clustering techniques like k-means/Gaussian Mixture Models allow us to group similar data points and rule learning without prespecified labels. Examples of application of these techniques include market segmentation and anomaly detection (Aissaoui et al., 2019). With the flexible nature of unsupervised learning, one gets useful insights with unstructured data. Techniques for Fine-Tuning Unsupervised Models for Classification Adapting unsupervised models for supervised tasks has recently been investigated, with k-means and GMM among them. Jain et al. (2010) showed that the k-means clusters could be labeled by attributing each k-means cluster to the majority class within each cluster. Thus, this adaptation substantially converts k-means into a supervised classifier. Huang et al. (2023) also showed that GMMs can probabilistically model data points as Gaussian components based on posterior probabilities, giving rise to soft classifications. These methods allow unsupervised models to be applied to datasets with overlapping features — a common scenario in real-world applications. However, as highlighted by Murphy (2012), the success of these adaptations relies on matching clusters with actual class boundaries, an issue sometimes tackled through preprocessing techniques, e.g., feature scaling and dimensionality reduction (like PCA). Jawad et al. (2023) also confirm the above by saying that GMMs are better than k-means on datasets with overlapped features. In contrast, k-means is better with well-separated clusters. It shows how knowing what the dataset looks like helps maximize performance. Use of Classifiers and Comparison of Performance Due to their high accuracy on labeled data, supervised classifiers, including Decision Trees, SVMs and Neural Networks, are widely used in machine learning. Although Decision Trees are often preferred for their interpretability, they may require proper pruning (Rokach & Maimon, 2014). SVMs are very good at high dimensional spaces but can be computationally intensive (Awad & Khanna, 2015). They are also neural networks that can capture complex non-linear relationships. Therefore, they are quite computationally expensive to train and need large labeled datasets (Goodfellow et al., 2016). However, in case of no labeled data, GMMs and k-means give alternatives. Sabri et al. (2024) showed that k means could be used to classify by assigning cluster labels to the most common class. GMMs extend this capability by incorporating probabilistic modeling, making them practical for datasets with overlapping classes (Liu & Vincent, 2017). Although unsupervised models, such as GMMs, suffer from their limitations, including sensitivity to initialization, under some circumstances, such as noisy or imbalanced datasets, they outperform traditional supervised methods (Khoei & Kaabouch, 2023). Overview and Analysis of Performance Measures Accuracy, Precision, Recall, F1 score, and ROC_AUC are used as metrics to evaluate classifier performance. Every metric serves a different purpose. For example, accuracy is a simple metric, but accuracy may have to perform better on imbalanced datasets (Hossin & Sulaiman, 2015). Precision is important and critical in applications like spam detection for minimizing false positives. Recall, or sensitivity, is also essential in life-critical domains like medical diagnosis. The F1 score balances Precision and Recall and is useful for measuring the imbalanced dataset. The main application of these metrics involves understanding model performance, but only when those metrics are used alongside confusion matrices to understand classification errors further. Adapting Unsupervised Models for Classification Tasks Adapting k means that GMM is also possible for supervised tasks. The idea was proposed by Sinaga and Yang (2020) to label clusters with the dominant class in each cluster, which turns the k-means problem into a classifier. As Scrucca (2021) showed, GMMs can assign data points as probabilistic assignments to Gaussian components. Preprocessing techniques, including scaling and dimensionality reduction, greatly benefit these adaptations in improving cluster alignment with real class boundaries (Jia et al., 2022). Also, Ahmed et al. (2020) have pointed out how feature engineering improves k-means and GMM performances, especially if noise class imbalance is present in the dataset. Together, these studies show the feasibility of unsupervised models in dealing with classification problems in the presence of limited labeled data. Methodology Data Collection The experimental framework demands a comprehensive set of datasets to assess the feasibility of adapting unsupervised models for supervised tasks. Past studies have suggested that datasets have varying characteristics, such as class proportions, feature densities, and noise levels (Jia et al., 2022). This guarantees that the models will be tested in conditions similar to the real world. Data from public repositories, such as the UCI Machine Learning Repository, will be sourced. Murphy (2012) defined best practices: each dataset will be preprocessed, including feature scaling and dimensionality reduction using Principal Component Analysis (PCA). The data will be split into training and testing sets using a standard 80 / 20 ratio to obtain consistent and reliable performance evaluations. Model Implementation In the study, the basic k means, and Gaussian Mixture Models (GMM) are adapted for classification tasks and compared with the traditional supervised classifiers (like Decision Tree, Support Vector Machine (SVM), and Neural Networks). 1 Unsupervised Models: 1 K-means: The clusters generated will be associated with class labels based on each cluster's most frequent class, as Jain et al. (2010) described. 2 GMM: A probabilistic framework will be used to assign data points to Gaussian components with the highest posterior probabilities, enabling soft classification, as highlighted by Scrucca (2021). 2 Supervised Models: 3 Decision Trees will serve as a baseline due to their interpretability and robustness in simple classification tasks (Rokach & Maimon, 2014). 4 Despite their computational complexity, SVMs will be employed for their high-dimensional data-handling capabilities (Awad & Khanna, 2015). 5 Neural Networks will be implemented to evaluate performance on complex, non-linear datasets (Goodfellow et al., 2016). Experimental Setup To ensure fair comparisons: * Cross-validation will be used for all models, dividing the training data into five folds for reliable performance estimates. * Performance will be evaluated using Accuracy, Precision, Recall, F1-score, and ROC-AUC (Hossin & Sulaiman, 2015). * Confusion matrices will be analyzed to identify classification errors and potential areas for improvement. Statistical Analysis The observed performance differences will be validated by statistical hypothesis testing. Paired t-tests or their non parametric equivalent will be conducted to assess the significance of performance model variations. This is consistent with the work of Ahmed et al. (2020), who highlighted the need to consider dataset-specific features in performance evaluations, for instance, dataset-specific properties like noise imbalance. Tools and Resources * Programming Language: Python, leveraging libraries like sci-kit-learn, numpy, and pandas. * Development Environment: Jupyter Notebooks for coding, documentation, and visualization. * Computational Resources: A workstation with sufficient computational power to handle model training and testing. Expected Outcomes 1 Unsupervised and supervised classifiers are analyzed, including their advantages and disadvantages. 2 A treatment of the kinds of data conditions under which unsupervised models can be fitted to solve supervised problems. 3 An introduction to the application of unsupervised classifiers in real-world problems. Timeline * Week 1 (November 7 –13): Systematically search for literature and identify and choose desirable datasets. * Week 2 (Nov 14 – 20): Use and modify generative and discriminative models. Create hierarchical models of the up and down typical traditional supervised models are: * Week 3 (November 21 – November 27): Train models on some data, perform tests, and learn the efficiency of utilizing a given data set. * Week 4 (Novembe...