Publication

Abstract : The growing occurrence of hate speech in the online political conversations is a great risk to the democratic values, yet the automated detection systems are still confused by the vagueness of the terms hate speech and offensive political satire. The current studies mainly focus on the establishment of new artificial intelligence architectures at the expense of data quality, which results in the creation of models with high false positive rates. In an attempt to close this gap, this study introduces a novel, publicly available dataset called PoliHate that has fine-grained features and consists of 6,766 samples of Reddit comments that were manually annotated and were collected during the turbulent post-2025 US Election period. We present a rigorous data processing pipeline that enables the Non-hate, Offensive, and Hate speech categories to be distinguished through an iterative annotation method. This change from textual instructions to a structured decision-tree schema enabled the authors to score a significant increase in Inter-Annotator agreement rate and elevate Krippendorff’s alpha from 0.347 to 0.517. To set a benchmark, we conducted an evaluation of Traditional Machine Learning, Deep Learning, and Transformer-based models. The experimental results showed that HateBERT achieved the best performance with a Macro-F1 score of 0.579. Moreover, even simple Logistic Regression models which used TF-IDF features were able to get better results than the more complex Deep Learning models like LSTM and CNN, which is indicative of the difficulties in generalizing about imbalanced and very nuanced datasets. The implications of these findings are that the keyword-based methods are limited and that the researchers have provided a highly accurate ground truth which is necessary for the development of hate speech detection systems that are aware of the context.