Distilling Semantic Knowledge via Multi-Level Alignment in TinyBERT-Based Language Models
- Authors
-
-
Tao Yang
-
Yu Cheng
-
Yijiashun Qi
-
Minggu Wei
-
- Abstract
-
This paper addresses the practical challenges of deploying large language models, particularly in terms of inference efficiency and resource consumption. It proposes an improved distillation framework. The method builds on the TinyBERT structure and introduces a multi-layer semantic alignment mechanism. This enhances the student model's ability to learn deep semantic and structural information from the teacher model. The approach jointly considers the transfer of output distributions, hidden layer representations, and attention matrices. A combined loss function is designed to optimize multiple distillation objectives. During training, the student model maintains a lightweight structure while effectively inheriting the expressive power of the teacher model. This improves its generalization and stability in multi-task scenarios. The experiments are conducted on the GLUE benchmark. Evaluation covers training dynamics, output distribution learning, task stability, and inference speed on low-resource devices. The results show that the proposed method outperforms mainstream distillation models across several metrics. It demonstrates strong compatibility, efficiency, and deployment adaptability. The findings further validate the effectiveness of multi-layer alignment strategies in improving the performance of compact language models. This provides a technical foundation for building high-performance, low-cost natural language processing systems.
- References
- Downloads
- Published
- 2025-05-30
- Issue
- Vol. 4 No. 5 (2025)
- Section
- Articles
- License
-

This work is licensed under a Creative Commons Attribution 4.0 International License.
Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.
How to Cite
Most read articles by the same author(s)
- Yu Cheng, Selective Noise Injection and Feature Scoring for Unsupervised Request Anomaly Detection , Journal of Computer Technology and Software: Vol. 3 No. 9 (2024)
- Minggu Wei, Federated Meta-Learning for Node-Level Failure Detection in Heterogeneous Distributed Systems , Journal of Computer Technology and Software: Vol. 3 No. 8 (2024)
- Tao Yang, Transferable Load Forecasting and Scheduling via Meta-Learned Task Representations , Journal of Computer Technology and Software: Vol. 3 No. 8 (2024)
