Structured Pruning-Driven High-Expressive Low-Rank Adapter Learning for Large Language Models

Authors
  • Ourong Lin

    University of California, Los Angeles, Los Angeles, USA

  • Gregory Brennan

    University of Southern California, Los Angeles, USA

  • Xiaosi Xu

    University of Southern California, Los Angeles, USA

Abstract

This paper addresses the challenge of balancing expressive power and resource overhead in efficient fine-tuning of large language model parameters. It proposes a structured pruning-driven high-expressive low-rank adapter learning method. By freezing the main model parameters, low-rank incremental updates are injected only into key linear transformation layers to achieve lightweight task adaptation. To overcome the expression bottleneck caused by fixed low rank, the method employs residual injection to controllably adjust hidden representations and introduces differentiable gating vectors in the low-rank channel dimension, transforming adapter capacity allocation into a learnable structural selection process, thereby achieving interpretable sparsity at the structural unit granularity. Furthermore, a unified optimization objective is constructed, consisting of the task objective, structured sparsity regularization, and budget constraints. Channel-level structured pruning is achieved using gating sparsity induction, and the overall trainable capacity is controlled by budget consistency constraints, forming a sparse, low-rank integrated parameterization and training mechanism. This framework organically integrates structured sparsity and low-rank representation, improving the effective expression density of the update subspace while maintaining parameter efficiency. It provides a controllable, well-structured, and easily implementable technical solution for adapter fine-tuning in resource-constrained deployments.

References
Cover Image
Downloads
Published
2025-08-30
Section
Articles
License
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.

How to Cite

Lin, O., Brennan, G., & Xu, X. (2025). Structured Pruning-Driven High-Expressive Low-Rank Adapter Learning for Large Language Models. Journal of Computer Technology and Software, 4(8). https://doi.org/10.5281/zenodo.21542856