Structured Pruning-Driven High-Expressive Low-Rank Adapter Learning for Large Language Models
- Authors
-
-
Ourong Lin
University of California, Los Angeles, Los Angeles, USA
-
Gregory Brennan
University of Southern California, Los Angeles, USA
-
Xiaosi Xu
University of Southern California, Los Angeles, USA
-
- Abstract
-
This paper addresses the challenge of balancing expressive power and resource overhead in efficient fine-tuning of large language model parameters. It proposes a structured pruning-driven high-expressive low-rank adapter learning method. By freezing the main model parameters, low-rank incremental updates are injected only into key linear transformation layers to achieve lightweight task adaptation. To overcome the expression bottleneck caused by fixed low rank, the method employs residual injection to controllably adjust hidden representations and introduces differentiable gating vectors in the low-rank channel dimension, transforming adapter capacity allocation into a learnable structural selection process, thereby achieving interpretable sparsity at the structural unit granularity. Furthermore, a unified optimization objective is constructed, consisting of the task objective, structured sparsity regularization, and budget constraints. Channel-level structured pruning is achieved using gating sparsity induction, and the overall trainable capacity is controlled by budget consistency constraints, forming a sparse, low-rank integrated parameterization and training mechanism. This framework organically integrates structured sparsity and low-rank representation, improving the effective expression density of the update subspace while maintaining parameter efficiency. It provides a controllable, well-structured, and easily implementable technical solution for adapter fine-tuning in resource-constrained deployments.
- References
- Downloads
- Published
- 2025-08-30
- Issue
- Vol. 4 No. 8 (2025)
- Section
- Articles
- License
-

This work is licensed under a Creative Commons Attribution 4.0 International License.
Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.
