Progressive Knowledge Distillation from RoBERTa to a Lightweight GRU-Based Student for Efficient Text Classification

• A non-Transformer student architecture (RoBERTa-initialized embeddings + BiGRU +
multi-head attention) that shows effective distillation is possible across architectural families, not only within the Transformer family.
• A progressive hybrid distillation objective combining soft, hard, sequence-level,
and representation-alignment losses under a dynamic, curriculum-based schedule, together with a three-stage training and Optuna-based hyperparameter search proto-
col.
• Rigorous cross-domain statistical validation: bootstrap AUC confidence intervals and DeLong significance testing on both a single-domain (IMDB) and a mixed multi-domain (IMDB + Yahoo + Yelp + custom) benchmark, quantifying not just the average accuracy–efficiency trade-off but its statistical stability.
• A quantified efficiency–accuracy operating point: 94.8% parameter reduction, 13.27× memory efficiency, and 2.71× latency reduction at >94% retained accuracy,
benchmarked directly against TinyBERT, DistilBERT, MobileBERT, FastBERT, and PKD.