{"id":607,"date":"2026-06-04T13:54:30","date_gmt":"2026-06-04T13:54:30","guid":{"rendered":"https:\/\/florentinudrea.net\/?p=607"},"modified":"2026-06-04T13:54:31","modified_gmt":"2026-06-04T13:54:31","slug":"meet-the-diffusion-policy-generative-ais-leap-into-the-physical-world","status":"publish","type":"post","link":"https:\/\/florentinudrea.net\/index.php\/2026\/06\/04\/meet-the-diffusion-policy-generative-ais-leap-into-the-physical-world\/","title":{"rendered":"Meet the Diffusion Policy: Generative AI\u2019s Leap into the Physical World"},"content":{"rendered":"\n

In the previous post<\/a>, we explored the rise of diffusion models in generative AI. The main takeaway was that instead of acting as simple “expected value” predictors, like classic neural networks do, diffusion models can map out complex, arbitrary distributions. This mathematical property is exactly why they are able to generate crisp, detailed images rather than averaging everything into a blurry blob.<\/p>\n\n\n\n

The robot learning community is no stranger to the “average the correct answers” kind of behavior either, especially in imitation learning paradigms (our way of teaching robots from expert data). Think of an autonomous car going straight towards a pole: its training dataset has seen expert drivers avoid the pole by going right or by going left<\/strong>. Both methods would be completely valid. A standard neural network, however, doesn’t see two distinct options. It tries to find a mathematical compromise between the two, averages the steering angles, and drives straight into the pole<\/strong> (not a fun ride, if you ask me<\/em>).<\/p>\n\n\n\n

Recently, the robot learning community realized that the core strength of diffusion models could fix this. By allowing the AI to evaluate multiple valid possibilities, without averaging them into a disastrous middle ground, a diffusion policy can confidently commit to just one, higher quality path<\/strong>. Going from generating digital art to steering a physical robot might sound like a wild leap, but it actually solves a massive headache in traditional AI control.<\/p>\n\n\n

\n
\"\"
A solution to the autonomous car example\u2026 maybe.<\/figcaption><\/figure>\n<\/div>\n\n\n
\n\n\n\n

<\/span>How to leverage Diffusion Models for Robot Learning<\/strong><\/span><\/h2>\n\n\n\n

Up until now we\u2019ve been talking about the world of pixels and image generation. But robots don’t output pixels, they use physical trajectories. So, how do we take an architecture built for generating DALL-E images and use it to control a thousands of dollars worth robotic arm?<\/p>\n\n\n\n

The conceptual leap is actually surprisingly elegant: we simply swap out the “image” for an “action trajectory<\/strong>“. Instead of denoising a 2D grid of RGB colors, the diffusion model denoises a sequence of motor commands over time, like joint angles, velocities, or the XYZ coordinates of a robotic gripper. The model still starts with pure, random noise, but as it iteratively refines that noise, it refines a smooth, physically viable path for the robot to follow. The mathematical process of “drawing a picture” becomes the exact same process as “planning a complex movement.”<\/p>\n\n\n\n

However, getting a robot to successfully execute these generated plans in the real world takes a bit more finesse than just prompting an image generator. The \u2018how\u2019 in training diffusion policies is rarely a single train run. It uses a two-stage approach: first building an imitation learning prior<\/strong> and then refining it via reinforcement learning<\/strong>.<\/p>\n\n\n\n

<\/span>Imitation Learning Prior<\/span><\/h3>\n\n\n\n

In the robotics community, we often use Imitation Learning (Behavior Cloning)<\/strong> to jumpstart a model. Since it is identical to supervised learning, the synergy with diffusion models is straightforward: the same noising and denoising process is used<\/strong>. The only difference here is that we are not generating images and we generate robot actions instead.<\/p>\n\n\n

\n
\"\"
Diffusion policy showing multimodal behavior wrt. other techniques in pushing a T block with an end effector (blue dot).
Results from “Diffusion Policy: Visuomotor Policy
Learning via Action Diffusion”<\/em>, Cheng et al. <\/figcaption><\/figure>\n<\/div>\n\n\n

<\/span>Reinforcement Learning Finetuning<\/span><\/h3>\n\n\n\n

Imitation is a great start, but it has a few problems. First, the performance will only be as good as the data it is provided<\/strong>. Second, counterintuitively, if the data is taken from an expert is never wrong, then the policy we are training will never know how to recover from the mistakes<\/strong> (which are basically ensured in the world of statistical learners, such as neural networks).<\/p>\n\n\n\n

Reinforcement learning overcomes this problems<\/strong>: it doesnt want to imitate anyone, it wants to maximize it\u2019s objective reward. Also, since it is closed-loop, it will face its errors and will learn how to recover from them.<\/p>\n\n\n\n

But here is where we hit a wall: diffusion models are inherently built to minimize a reconstruction loss<\/strong>, they are trained via maximum likelihood to make their generated outputs look exactly like the training data. Reinforcement Learning<\/strong>, on the other hand, doesn’t care about mimicking data (it doesn’t even have target data)<\/strong>: it cares about maximizing a reward, and the optimal behavior to achieve that reward might not even exist in any fixed dataset.<\/p>\n\n\n\n

Merging these two worlds is not trivial<\/strong>. To train a diffusion model with RL using standard gradients, you would theoretically need to backpropagate the reward signal through the entire reverse diffusion chain. That means calculating gradients through dozens or hundreds of sequential denoising steps, which is a total nightmare for memory and stability.<\/p>\n\n\n\n

\n\n\n\n

<\/span>The Reinforcement Learning Mental Shift<\/strong><\/span><\/h2>\n\n\n\n

To grasp this fully, a mental shift helps. We typically view Reinforcement Learning strictly as reward optimization. While true, RL is fundamentally a mathematical framework for optimizing any non-differentiable objective function<\/strong>.<\/p>\n\n\n\n

Standard deep learning requires smooth, continuous math to calculate gradients and update weights. RL bypasses this constraint. It observes final outcomes and increases the probability of the network outputting actions that led to those results. We usually define these results as task success, but the target can be anything. For example, you could use RL to minimize a robot arm’s total energy consumption or maximize the physical smoothness of a trajectory.<\/p>\n\n\n\n

With diffusion models, the optimization changes. We are not just updating the network to output a specific final action. We are optimizing the intermediate denoising steps the network takes to build that action.<\/strong><\/p>\n\n\n\n

<\/span>How to train Diffusion Policies with RL ?<\/span><\/h3>\n\n\n\n

So, how do we combine diffusion’s “imitate the data” with RL’s “maximize the score”? Standard RL expects a direct, one-step link between seeing a state and taking an action, but diffusion takes dozens of intermediate “denoising” steps. Here are three clever ways researchers are bridging this gap (click on the title to open explanation<\/em>):<\/p>\n\n\n\n

\n
\n