Publication

Abstract : Feature engineering is critical for improving machine learning performance (ML), especially when handling categorical data. Traditional encoding methods, such as one-hot and label encoding, often result in challenges like high dimensionality and loss of category significance. To address these limitations, this study introduces a novel Chi-Square Target Encoding (CSTE) approach, which transforms categorical variables into numerical representations by leveraging the chi-square statistic to evaluate the association between categories and the target variable, preserving information effectively. Unlike conventional techniques, CSTE uses Chi-square test values to generate target-based encoded representations, ensuring reliable transformation without information loss. Comprehensive empirical evaluations were conducted on various datasets for binary classification, showcasing CSTE’s superiority over methods like regularized target encoding, basic target encoding, and fuzzification. A case study on real-world sensor data further validated its efficiency and scalability for large-scale data-driven applications. The proposed CSTE method achieved an average increment rate of 3.9447%, surpassing one-hot and fuzzification (1.0430%) and regularized target encoding (1.8308%). Classification outcomes demonstrated F1-scores exceeding 0.90 and AUC values nearing 0.99 across diverse datasets, highlighting its robustness. Furthermore, the reduced dimensionality significantly enhanced inference time while maintaining high accuracy. The CSTE method offers a robust framework for categorical data transformation, addressing limitations of traditional encoding techniques. It improves the interpretability and efficiency of categorical data representation, boosting ML performance. This innovative approach is well-suited for applications across various domains involving categorical data.