Privacy-Oriented Text Generation in LLMs via Selective Fine-Tuning and Semantic Attention Masks

Authors
  • Renhan Zhang

Abstract

This paper addresses the issue of sensitive entity exposure in text generation by large language models. It proposes a control method that combines selective fine-tuning with a semantic-guided masking strategy. First, a parameter selection mechanism is designed. By analyzing gradient responses in sensitive entity contexts, the method identifies a subset of parameters strongly related to sensitive expressions. Only this subset is updated during fine-tuning, allowing for fine-grained behavioral adjustment. Second, a semantic-guided masking strategy is introduced. It uses the model's semantic features to construct a probabilistic mask. This mask intervenes at the attention mechanism level to reduce the model's response strength to sensitive token positions, lowering the likelihood of explicit expression. The experimental section validates the effectiveness of the method across multiple dimensions. These include comparisons with mainstream fine-tuning strategies, ablation studies, intervention effects across structural layers, sensitivity recovery tests, and training step analysis. Results show that the proposed method significantly reduces the exposure rate of sensitive entities while maintaining or even improving contextual coherence and generation quality. It demonstrates strong practicality and stability. This method provides an effective path for secure generation control in large models. It suppresses potential privacy risks while preserving the model's expressive capabilities.

References
Cover Image
Downloads
Published
2025-08-30
Section
Articles
License
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.

How to Cite

Zhang, R. (2025). Privacy-Oriented Text Generation in LLMs via Selective Fine-Tuning and Semantic Attention Masks . Journal of Computer Technology and Software, 4(8). https://doi.org/10.5281/zenodo.17074568