Dynamic Token Pruning via Semantic-aware Distillation for Efficient Transformer Compression
- Authors
-
-
Callum Petros
-
- Abstract
-
Transformer-based models have achieved state-of-the-art performance across various domains but often suffer from high computational cost due to the quadratic complexity of self-attention. In this paper, we introduce a novel Dynamic Token Pruning framework that integrates Semantic-aware Distillation to compress large-scale Transformers effectively. Our method identifies and removes redundant tokens at each layer based on their contribution to final predictions, guided by a teacher model trained on full sequences. A lightweight gating module evaluates token importance dynamically, ensuring adaptive pruning without retraining from scratch. Experiments on image classification (DeiT, ViT) and natural language understanding tasks (BERT, RoBERTa) show that our approach achieves up to 50% reduction in FLOPs with less than 1% drop in accuracy. Furthermore, ablation studies validate the effectiveness of semantic guidance in retaining critical contextual information. This work provides a scalable and model-agnostic solution to deploy Transformer models on resource-constrained devices.
- References
- Published
- 2024-11-30
- Issue
- Vol. 3 No. 8 (2024)
- Section
- Articles
- License
-

This work is licensed under a Creative Commons Attribution 4.0 International License.
Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.
