Structured Preference Modeling for Reinforcement Learning-Based Fine-Tuning of Large Models
- Authors
-
-
Lin Zhu
-
Fan Guo
-
Guohui Cai
-
Yumeng Ma
-
- Abstract
-
This paper aims to explore how preference modeling can enhance policy optimization efficiency and behavior controllability during reinforcement learning fine-tuning of large models. To address the limitations of traditional RLHF methods in modeling human feedback and guiding policy learning, we propose a strategy optimization framework that integrates a multi-scale preference modeling mechanism. The proposed method first constructs a structured preference scoring function from human feedback data to approximate reward signals. It then combines this with a policy gradient approach to guide the fine-tuning of language models, enabling effective alignment between model behavior and human preferences. The experimental section evaluates the performance of different preference modeling strategies on multiple natural language generation tasks. A comparative analysis is conducted across several dimensions, including accuracy, preference alignment, convergence speed, and training stability. Results show that the proposed method achieves better overall performance than existing approaches. It demonstrates strong capability in modeling preferences and improving fine-tuning effectiveness.
- References
- Downloads
- Published
- 2025-04-30
- Issue
- Vol. 4 No. 4 (2025)
- Section
- Articles
- License
-

This work is licensed under a Creative Commons Attribution 4.0 International License.
Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.
How to Cite
Most read articles by the same author(s)
- Fan Guo, Lin Zhu, Yichen Wang, Guohui Cai, Perception-Guided Structural Framework for Large Language Model Design , Journal of Computer Technology and Software: Vol. 4 No. 5 (2025)
