Role Mutual Information Play in Feature Selection

Mutual Information for Feature Selection
Name
Institutional Affiliation
Mutual Information for Feature Selection
While the model selection is critical in learning signals from the provided data, providing the right variables or data is of utmost importance. In machine learning, the model building requires the construction of relevant features or variables through feature engineering, and the resulting data set can then be employed as a statistical input to train a model. While these models are often assumed to be sophisticated and smart algorithms, they are easily fooled by unnecessary clutter and dependencies in the data. Data scientists often make the signals to be easily identifiable by performing feature selection, which is a necessary step in data pre-processing (Huijskens, 2017). According to Zhou, Wang, and Zhu (2022), feature selection is a fundamental pre-processing step in machine learning as it selects only the crucial features by eliminating redundant or irrelevant features from the primary dataset. Data scientists use algorithms that maximize only relevant information while minimizing unnecessary or redundant information. One of these techniques that have been adopted by machine learning experts and data scientists is mutual information feature selection. In this algorithm, the in-filter feature selection approach is used in assessing the relevancy of a subset of features to predict the largest variable as well as the redundancy based on other variables. Nevertheless, Beraha et al. (2019) note that the existing algorithms are often heuristic and fail to provide any guarantee that they will resolve a proposed problem. This limitation has motivated the authors to propose a novel way of observing the theoretical results that indicate conditional mutual information may occur naturally when bunding the ideal regression or classification errors that are achieved by various features or a subset of features. This paper reports on the progress of my project on mutual information for feature selection and particularly presents what, why, and how feature selection and the roles of mutual information in feature selection.
What? Why? And How? Feature Selection
In machine learning, it is almost unlikely that all variables in a dataset will be useful in building real-life models, and the addition of any redundant or unnecessary variables will ultimately reduce the capacity of the model to generalize. Besides, redundant variables also reduce the accuracy of classifiers used in such models and may result in increasing the model complexity. The main purpose of feature selection is, therefore, to find the best set of features allowing data scientists to build useful models that are studied in a phenomenon. Implementing feature selection takes any of the two approaches: supervised and unsupervised techniques or learning. In supervised learning, labeled data is used in identifying the relevant features to increase the efficiency of supervised models like regression or classification, while unsupervised techniques use unlabelled data (Gupta, 2020). These approaches are broadly classified into four different methods, including filter, wrapper, embedded, and hybrid methods. In filter methods, the models select the intrinsic properties of the measured features using univariate statistics instead of performing cross-validation. Filter methods are known to be faster and require less computational power compared to the wrapper methods. Wrapper techniques, on the other hand, require the models to search the spaces for every possible feature subset while assessing their qualities through learning and evaluation of the classifiers with the feature subsets. In embedded methods, the benefits of the filter and wrapper method are combined through the inclusion of interactive features while maintaining reasonable computational demands. These methods are often iterative in terms of taking care of every iteration of the process of training models and will carefully extract the features that have an important contribution to the model training for each iteration.
Roles of Mutual Information in Feature selection
Mutual information (MI) refers to a measure of statistical independence with two main properties: measuring any kind of relationship between random variables, even non-linear relationships, and having invariance under transformations in terms of feature space which are differentiable and invertible such as rotations, translations, or any form of transformation. MI preserves the order of the original elements of the feature vectors. The pioneering work of Battiti introduced the definition of the feature selection problem as a step aimed at selecting the k most relevant variables amongst m variables from an original feature set, i.e., k < m. Battiti then proposed an alternative greedy selection of a single feature each at a time in evaluating the combinatorial explosion of all the features in the original feature set. He then arrived at four assumptions: features can only be classified in terms of either redundancy or relevancy; heuristic functional approaches are employed when selecting features and allow the controlling of the trade-off between redundancy and relevancy; g...