A Super-Convergent Knowledge Distillation Framework for Domain-Specific Text Classification

Large language models have pushed the recent-most
in Natural Language Processing (NLP), but are often too high
cost to deploy in safety-critical, domain-specific use cases such
as biomedical text processing, where efficiency is essential. This paper proposes a knowledge distillation framework for learning
lightweight and effective models. The framework rests on two
key methodological pillars, firstly a multi-teacher ensemble that
supplies diverse, strong knowledge representations to the student,
and secondly an adaptive, multi-component loss function that
shifts focus among several learning signals during training. The
overall objective is to achieve what we call super-convergence,
a setting where a compact student model retains much of its
teachers’ capability while remaining highly competitive with a
much larger ensemble. Experiments on a biomedical sentence
classification task (PubMed 20k RCT) show that a distilled
DistilBERT student with about 66M parameters achieves 0.8659
accuracy and 0.8623 weighted F1, coming within roughly 0.9
F1 points of a ∼287M-parameter heterogeneous BERT-family
ensemble. In practice, this corresponds to a more than 4.3×
reduction in parameter count and substantially faster inference,
with only a marginal loss in predictive performance. These
results highlight a practical direction for building resource
conscious yet high-performing AI systems in specialized domains.