An Optimization Perspective on Guidance for Fine-Tuning Diffusion Models
Diffusion models have achieved state-of-the-art performance in generative modeling, with their power significantly enhanced by fine-tuning toward task-specific objectives. However, existing fine-tuning methods often lack a principled foundation and offer limited performance guarantees. In this talk, we present a mathematical framework for understanding guidance-based diffusion model fine-tuning, providing a systematic perspective on its optimization properties and algorithmic design.
We abstract task-specific objectives as a reward function and fine-tuned diffusion models aim to maximize the reward by generating solutions. In the offline setting, we show that guidance enables high-quality sample generation, achieving optimality akin to off-policy bandit algorithms. In the online setting with real-time feedback, we establish a strong connection between guided diffusion and optimization. Specifically, gradient guidance-based diffusion effectively samples solutions to a regularized optimization problem, where the regularization arises from logged data. As for guidance design, directly bringing in the gradient of the reward function as guidance would jeopardize the structure in generated samples. We investigate a modified form of gradient guidance based on a forward prediction loss, which provably preserves the latent structure. We further consider an iteratively fine-tuned version of gradient-guided diffusion where guidance and score network are both updated with newly generated samples. This process mimics a first-order optimization iteration in expectation, for which we prove O(1/K) convergence rate to the global optimum when the objective function is convex.
Bio: Minshuo Chen is an assistant professor with the Department of Industrial Engineering & Management Sciences at Northwestern University. He was an associate research scholar with the Department of Electrical and Computer Engineering at Princeton University from 2022 to 2024. He completed his Ph.D. from the School of Industrial and Systems Engineering at Georgia Tech, majoring in machine learning. His research focuses on developing principled methodologies and theoretical foundations of deep learning, with a particular interest in 1) generative models including diffusion models, 2) foundations of machine learning, such as optimization and sample efficiency, and 3) reinforcement learning.