SimSighs: Benchmarking and Explaining Transformer Models for Bengali Telecom Complaints

The increasing amount of customer feedback in
Bengali presents a significant problem for telecom operators.
Analyzing and classifying such data takes a lot of time and does
not always produce accurate results. This article introduces
SimSighs, an improved and relabeled benchmark dataset for
Bengali telecom complaint classification. We aim to address the
problems caused by inconsistencies in existing public datasets,
merge classes with similar meanings, and balance the number of
samples across all classes. We also evaluate twelve different
models, ranging from traditional machine learning to deep
learning and state-of-the-art transformer architectures. Our
experiments demonstrate that models based on transformer
architecture, especially Bangla-BERT, achieve the best results
in terms of F1-score, reaching 92.22%. We apply Explainable
AI (XAI) to understand which parts of the input text contribute
most to a model’s decision. The visualization technique, Layer
Integrated Gradients, helps identify patterns and words that a
model uses to classify a complaint based on its category. Overall,
our research emphasizes the importance of high-quality datasets
and the application of XAI methods for developing dependable
and accurate NLP systems for customer feedback analysis.