Explainable and Generalizable Deepfake Detection for Vishing Attack Recognition

Artificial Intelligence (AI) has come a long way in
mimicking human voice to the extent that it cannot be identified
without extended effort. As a result, AI-generated vishing attacks
have increased threat, as humans mostly get deceived by the
highly human-alike AI-generated voice. Proposed study aims to
build a generalizable and explainable framework that can identify
both deepfake and scam intent. For deepfake detection, a voting
ensemble containing Hubert-base and WavLM-large is zero-shot
tested on the out-of-domain for generalization. Additionally, for
scam-intent classification, DistilBERT classifies scam intents for
transcripts generated via Whisper speech-to-text. Both branches
produce a combined result, which can be from three classes: AI-
Legit, AI-Scam, and Human-Legit, utilizing a rule-based joint
probability threshold of 0.5. Shortcut learning is evaluated using
Generalization, Robustness, and Stability Index (GRSI), a six-test
perturbation battery, and silence diagnostics. Dual-level SHAP
is used to provide acoustic and word-level explanation. During
held-out internal testing, DistilBERT showed 99% accuracy with
0.98 recall, while the ensemble for deepfake detection showed
90% accuracy after soft voting. Hence, the proposed framework
showcases generalization, and explainability in identifying AI-
enabled attacks while decreasing the risk of shortcut learning