Publication

Abstract : Efficient encoding of categorical data is essential for enhancing the performance of machine learning (ML) models, statistical methods, particularly when handling high-cardinality features and complex relationships between variables. Traditional encoding methods have notable limitations. One-hot encoding (OHE) often results in high-dimensional feature spaces, reducing model efficiency and predictive accuracy; label encoding (LE) can introduce misleading ordinal relationships; binary encoding (BE) may create excessively large feature representations for high-cardinality categories; weight of evidence (WoE) is prone to overfitting when rare categories are present; and entity embedding (EE) is less interpretable, and may perform poorly on unseen categories. To address these challenges, this paper introduces weight of evidence–entity embedding encoding (WoEEE), a novel encoding technique that integrates the strengths of both WoE and EE. WoE, a statistical approach, encodes categorical variables based on their influence on the target variable, while EE, a DL-based technique, transforms categories into continuous embeddings to capture complex, nonlinear relationships. The proposed WoEEE encoding method first applies WoE to identify and encode significant categories, followed by EE to generate low-dimensional embeddings. Comprehensive empirical evaluations were conducted on various binary and multiclass classification datasets to assess the effectiveness of the proposed WoEEE encoding method. The results, obtained using classifiers such as random forest (RF), Decision Tree (DT), logistic regression (LR), eXtreme gradient boosting (XGBoost), and light gradient boosting machine (LightGBM), demonstrate that WoEEE improves performance over the conventional encoding techniques as well as advanced methods including fuzzification-based encoding, deep-learned embedding, deep feature embedding, and regularized target encoding. On the Adult dataset, WoEEE achieves an F1-score of 0.98, compared to 0.85 for OHE and 0.88 for LE. Additionally, WoEEE improves computational efficiency by reducing feature dimensionality by up to 70%, without compromising accuracy. Furthermore, WoEEE exhibits strong scalability, efficiently processing large, high-cardinality datasets. The proposed work introduces WoEEE as a powerful, interpretable, and scalable solution for categorical data encoding, with potential applications across diverse fields, including healthcare, finance, and marketing.