FEATURE SELECTION FOR HIGH-DIMENSIONAL TEXT CLASSIFICATION USING TF-IDF REPRESENTATIONS
Keywords:
text classification, TF-IDF, feature selection, chi-square, mutual information, sparse machine learning, interpretable NLPAbstract
TF-IDF vectorization remains a strong and transparent representation for classical text classification, but it often creates sparse feature spaces with thousands of terms and n-grams. This study evaluates whether feature selection can improve classification quality, reduce training cost, and support term-level interpretability in high-dimensional NLP pipelines. A reproducible Python experiment was implemented with a Pang and Lee sentence-polarity sample containing 200 labelled movie-review snippets. After text cleaning and TF-IDF vectorization with word unigrams, bigrams, and trigrams, the training matrix contained 6,440 sparse features. Five selection strategies were compared: Chi-square, Mutual Information, ANOVA F-test, L1 Logistic Regression coefficients, and Linear SVM coefficients. Four classifiers were evaluated: Logistic Regression, Linear SVM, Multinomial Naive Bayes, and Random Forest. The number of selected features was varied from 100 to 5,000 and compared with the all-feature baseline. The best observed configuration was ANOVA F-test with k=5000 and Linear SVM, obtaining accuracy=0.6000, macro F1=0.6000, and weighted F1=0.6000. The best all-feature baseline achieved macro F1=0.5489. Although the dataset is intentionally small and the results should not be generalized without larger corpora, the experiment demonstrates that feature selection can improve compactness, reveal discriminative terms, and provide a reproducible diagnostic tool for sparse text-classification workflows.References
[1] Q. Li, H. Peng, J. Li, C. Xia, R. Yang, L. Sun, P. S. Yu, and L. He, “A Survey on Text Classification: From Traditional to Deep Learning,” ACM Transactions on Intelligent Systems and Technology, vol. 13, no. 2, 2022. DOI: 10.1145/3495162.
[2] K. Taha, “A comprehensive survey of text classification techniques,” 2024. Available via ScienceDirect.
[3] N. Pudjihartono et al., “A Review of Feature Selection Methods for Machine Learning-Based Disease Risk Prediction,” Frontiers in Bioinformatics, 2022.
[4] “Review of feature selection methods for text classification,” International Journal of Advanced Computer Research, 2020.
[5] G. Kou et al., “Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods,” Applied Soft Computing, 2020.
[6] H. Mamdouh Farghaly, “A high-quality feature selection method based on frequent and correlated items for text classification,” Soft Computing, 2023.
[7] C. Tang, “Review on Application of Chi-square Statistic in Text Classification in Recent Five Years,” 2024.
[8] scikit-learn developers, “TfidfVectorizer,” scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
[9] scikit-learn developers, “Feature selection,” scikit-learn documentation. https://sklearn.org/stable/modules/feature_selection.html
[10] scikit-learn developers, “mutual_info_classif,” scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html
[11] M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, and P. Sen, “A Survey of the State of Explainable AI for Natural Language Processing,” arXiv:2010.00711, 2020.
[12] S. Luo, H. Ivison, C. Han, and J. Poon, “Local Interpretations for Explainable Natural Language Processing: A Survey,” arXiv:2103.11072, 2021.
[13] B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” ACL, 2005.
[14] B. Pang and L. Lee, “Movie Review Data,” Cornell University. https://www.cs.cornell.edu/people/pabo/movie-review-data/