STABILITY ANALYSIS OF FEATURE SELECTION ALGORITHMS UNDER DIFFERENT DATA SPLITS AND RANDOM SEEDS
Keywords:
feature selection stability, Jaccard similarity, robust machine learning, medical AI, random seeds, scikit-learn pipelinesAbstract
Feature selection is widely used to improve interpretability, reduce dimensionality and support robust deployment of machine-learning models, especially in domains such as medical artificial intelligence where explanations are expected to be reproducible. However, predictive performance alone does not reveal whether an algorithm selects a stable and interpretable subset of features. This study evaluates the stability of seven feature-selection algorithms under changes in random seed, train/test split and cross-validation folds: ANOVA F-test, mutual information, chi-square, recursive feature elimination, L1-based SelectFromModel, Random Forest importance and Extra Trees importance. Experiments were conducted on four real open classification data sets from scikit-learn (Breast Cancer Wisconsin, Wine, Iris and Digits) and one explicitly synthetic data set containing informative, redundant and noisy features. For each data set and algorithm, ten random seeds were used. Accuracy, precision, recall, F1-score, ROC-AUC, pairwise Jaccard similarity between selected feature subsets and feature-selection frequencies were computed. The results show that high predictive performance does not necessarily imply stable feature selection. L1-based selection achieved the highest mean ROC-AUC (0.9819) and F1-score (0.9380), while ANOVA F-test achieved the highest seed-level Jaccard stability (0.9126). RFE with a linear classifier produced high mean ROC-AUC (0.9816) but the lowest average stability (0.6722). The Spearman correlation between mean ROC-AUC and seed-level Jaccard similarity was close to zero (-0.0252), indicating that performance and stability capture different dimensions of model quality.References
[1] Büyükkeçeci, M.
Okur, M.C. A Comprehensive Review of Feature Selection and Feature Selection Stability in Machine Learning. Gazi University Journal of Science 2023, 36(4), 1506-1520. https://doi.org/10.35378/gujs.993763
[2] Khaire, U.M.
Dhanalakshmi, R. Stability of feature selection algorithm: A review. Journal of King Saud University - Computer and Information Sciences 2022, 34(4), 1060-1073. https://doi.org/10.1016/j.jksuci.2019.06.012
[3] Łukaszuk, T.
Krawczuk, J.
Żyła, K.
Kęsik, J. Stability of Feature Selection in Multi-Omics Data Analysis. Applied Sciences 2024, 14(23), 11103. https://doi.org/10.3390/app142311103
[4] Lazebnik, T.
Rosenfeld, A. A new definition for feature selection stability analysis. Annals of Mathematics and Artificial Intelligence 2024, 92, 753-770. https://doi.org/10.1007/s10472-024-09936-8
[5] Barbieri, M.C.
Grisci, B.I.
Dorn, M. Analysis and comparison of feature selection methods towards performance and stability. Expert Systems with Applications 2024, 249, 123667. https://doi.org/10.1016/j.eswa.2024.123667
[6] Gundersen, O.E.
Coakley, K.
Kirkpatrick, C.
Gil, Y. Sources of Irreproducibility in Machine Learning: A Review. arXiv 2022. https://doi.org/10.48550/arXiv.2204.07610
[7] Amir, S.
van de Meent, J.-W.
Wallace, B.C. On the Impact of Random Seeds on the Fairness of Clinical Classifiers. arXiv 2021. https://doi.org/10.48550/arXiv.2104.06338
[8] Williams, N.
Schuler, A. Improving reproducibility by controlling random seed stability in machine learning. arXiv 2026. https://doi.org/10.48550/arXiv.2604.17694
[9] de Hond, A.A.H.
Leeuwenberg, A.M.
Hooft, L.
Kant, I.M.J.
Nijman, S.W.J.
van Os, H.J.A.
Aardoom, J.J.
Debray, T.P.A.
Schuit, E.
van Smeden, M.
Reitsma, J.B.
Steyerberg, E.W.
Chavannes, N.H.
Moons, K.G.M. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. npj Digital Medicine 2022, 5, 2. https://doi.org/10.1038/s41746-021-00549-7
[10] Scikit-learn Developers. Feature selection. scikit-learn User Guide, version 1.9.0 documentation. Accessed 2 July 2026. https://scikit-learn.org/stable/modules/feature_selection.html
[11] Scikit-learn Developers. sklearn.pipeline.Pipeline. scikit-learn API Reference, version 1.9.0 documentation. Accessed 2 July 2026. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html
[12] Scikit-learn Developers. Toy data sets: load_breast_cancer, load_wine, load_iris and load_digits. scikit-learn API Reference, version 1.9.0 documentation. Accessed 2 July 2026. https://scikit-learn.org/stable/datasets/toy_dataset.html
[13] MDPI. Instructions for Authors: Information. Accessed 2 July 2026. https://www.mdpi.com/journal/information/instructions
[14] Jiang, Y.
Wang, X.
Liang, C.
Jiang, B.
Li, H. Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data. arXiv 2020. https://doi.org/10.48550/arXiv.2012.00001
[15] Nogueira, S.
Sechidis, K.
Brown, G. On the stability of feature selection algorithms. Journal of Machine Learning Research 2018, 18(174), 1-54.
[16] Pedregosa, F.
Varoquaux, G.
Gramfort, A.
Michel, V.
Thirion, B.
Grisel, O.
Blondel, M.
Prettenhofer, P.
Weiss, R.
Dubourg, V.
Vanderplas, J.
Passos, A.
Cournapeau, D.
Brucher, M.
Perrot, M.
Duchesnay, É. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12, 2825-2830.