{"id":572,"date":"2026-04-17T16:19:49","date_gmt":"2026-04-17T16:19:49","guid":{"rendered":"https:\/\/florentinudrea.net\/?p=572"},"modified":"2026-04-17T16:22:53","modified_gmt":"2026-04-17T16:22:53","slug":"diffusion-models-the-secret-sauce-of-generative-ai","status":"publish","type":"post","link":"https:\/\/florentinudrea.net\/index.php\/2026\/04\/17\/diffusion-models-the-secret-sauce-of-generative-ai\/","title":{"rendered":"Diffusion Models: The Secret Sauce of Generative AI"},"content":{"rendered":"\n

You\u2019ve probably seen the absolute explosion of AI-generated art over the last few years. Whether it\u2019s a hyper-realistic image of a dog riding a skateboard on Mars or a stunning digital painting: the internet is flooded with this stuff (especially my Instagram reels). The engine under the hood of all these incredible image generators (like Midjourney, DALL-E, Stable Diffusion and more recently NanoBanana) is something called a diffusion model<\/strong>. They work, quite literally, by taking an image of full of noise and slowly “denoising” it step-by-step until the image appears.<\/p>\n\n\n\n

But why did diffusion completely take over the image generation world? The secret sauce is in how it handles complex distributions. Older image generators were tricky to train and often produced blurry or weirdly distorted results when asked to do something complicated. By design, classic neural networks are expected value<\/em> approximators<\/em>: they learn the mean of the distribution they are trying to model. This is why previously, when a face was image-generated, it looked very generic, almost like it was a blend of all the face images it has seen: that\u2019s because it was<\/strong>, outputting the mean of all faces it has seen it\u2019s the way to minimize the global loss during training.<\/p>\n\n\n\n

Diffusion models, which work by taking random noise and slowly “denoising” it step-by-step, changed the game. They are stable to train, can capture fine detail, and (most importantly) they handle multiple valid options<\/strong>. If you ask a diffusion model for a picture of a house, it doesn’t just mathematically average every house it has ever seen into one blurry, generic blob. It commits to one<\/em> distinct, high-quality sample.<\/p>\n\n\n

\n
\"\"
Diffusion process. Source geeksforgeeks.org<\/figcaption><\/figure>\n<\/div>\n\n\n

Unpacking the Core Concept: What is a Diffusion Neural network?<\/h2>\n\n\n\n

To move from the specific application in robotics back to the broader architecture, we have to look at why Diffusion Neural Networks<\/strong> are fundamentally different from the “input-in, output-out” models we\u2019ve used previously. In a standard deep learning model, the goal is to find a direct mapping function. If you want to generate a specific output, the network tries to jump from the input to the final result in a single forward pass. This usually works for one-to-one tasks, like classification (where there same input should lead to the same output: an image of a flower will\/should always be classified as a flower). In one-to-many problems, like Generative AI<\/strong>, where ‘generating a flower’ has multiple correct outputs, the mapping mechanism of standard deep learning models falls apart.<\/p>\n\n\n\n

The Score-Based Perspective<\/strong> Diffusion models change the objective from mapping<\/strong> to gradient following<\/strong>. Instead of learning what a “perfect” data point looks like, the network learns the score function,<\/strong> the gradient of the log-probability of the data.<\/p>\n\n\n\n

Think of the data manifold (the collection of all valid, high-quality outcomes) as a series of mountain peaks. In this metaphor, the higher the elevation, the more “realistic” or “correct” the data is. The valleys below represent the infinite ways to produce noise or nonsensical results. Traditional Models<\/strong> tries to “teleport” from an input directly onto a peak. However, if the training data contains two different peaks (like a car turning left and a car turning right), the model\u2019s math forces it to aim for the “average” coordinates between them. It tries to land in the middle of the two mountains, only to realize there is no ground there. Diffusion Models<\/strong> don’t try to leap to the summit in one go. Instead, it learns the score function<\/strong>: a mathematical “map of the slopes.” It studies the entire landscape and learns which way is “up” from any given point in the fog.<\/p>\n\n\n\n

Handling the “One-to-Many” Problem<\/strong> The most significant advantage of this architecture is its inherent ability to handle one-to-many<\/strong> relationships. In standard regression, if the input X<\/mi>X<\/annotation><\/semantics><\/math> could result in either Y<\/mi>1<\/mn><\/msub>Y_1<\/annotation><\/semantics><\/math> or Y<\/mi>2<\/mn><\/msub>Y_2<\/annotation><\/semantics><\/math>, the model’s loss function (like Mean Squared Error) will force it to predict Y<\/mi>1<\/mn><\/msub>+<\/mo>Y<\/mi>2<\/mn><\/msub><\/mrow>2<\/mn><\/mfrac>\\frac{Y_1+Y_2}{2}<\/annotation><\/semantics><\/math>. Diffusion networks avoid this because they are probabilistic samplers<\/strong>. They don’t predict a value; they model a distribution. By starting from a different noise each time, the same network can walk its way up different mountains (valid regions of the data). It treats the complexity of the real world not as noise to be averaged out, but as a landscape of valid peaks to be explored.<\/p>\n\n\n
\n
\"Example
Example of the diffusion process going towards the targets (peaks in our analogy).<\/figcaption><\/figure>\n<\/div>\n\n\n
\n\n\n\n

Speeding up Diffusion Models<\/h2>\n\n\n\n

Denoising Diffusion Probabilistic Models (DDPMs): The Stochastic Random Walk<\/h4>\n\n\n\n

DDPMs are the architecture that started the current generative AI boom. They operate as a Stochastic Differential Equation (SDE)<\/strong>.<\/p>\n\n\n\n

In a DDPM, the forward process destroys an image by injecting Gaussian noise step-by-step. The neural network is trained to estimate the “score” (the gradient of the log probability) to reverse this process.<\/p>\n\n\n\n

Because the noise injection is random (stochastic), the mathematical path from a clean image to pure noise is<\/strong> essentially Brownian motion, it is jagged, curved, and highly chaotic<\/strong>. When the neural network tries to reverse this process, it is forced to navigate this curved path.<\/p>\n\n\n\n

If the mathematical “step size” it takes is too large, it will overshoot the curve and step off the data manifold, resulting in a distorted, unrecognizable image. Therefore, the model is trapped into taking hundreds or thousands of tiny steps<\/strong> (neural network evaluations)<\/strong> to safely traverse the curved probability space. This is what makes traditional diffusion so computationally expensive and slow.<\/p>\n\n\n\n

Flow Matching: Deterministic Vector Fields<\/h4>\n\n\n\n

Flow Matching abandons the stochastic random walk entirely. Instead, it frames the generation process as an Ordinary Differential Equation (ODE)<\/strong>. It treats the noise and the target data not as a sequence of corrupted images, but as two distinct distributions of particles.<\/p>\n\n\n\n

Imagine a solid object made of sand (the data distribution). Now imagine you put a tiny explosive at the middle of it and detonate it (noising): it will become a cloud of sand grains (the Gaussian noise distribution). When accounting for air turbulence, the direction in which each grain of sand was blown away is noisy and unpredictable. On the other hand, if you ignore the turbulence of the air, every grain of sand has a distinct direction<\/strong> in which it was blown away. That is the main difference between diffusion models and flow matching models: the former model noisy and stochastic paths, the latter model clear paths.<\/p>\n\n\n\n

Flow Matching trains a neural network to act as a Velocity Vector Field<\/strong>, denoted as v<\/mi>t<\/mi><\/msub>(<\/mo>x<\/mi>)<\/mo><\/mrow>v_t(x)<\/annotation><\/semantics><\/math>. The network does not try to guess “how much noise to remove.” Instead, it looks at a particle at position x<\/mi>x<\/annotation><\/semantics><\/math> at time t<\/mi>t<\/annotation><\/semantics><\/math> and outputs a physical velocity vector: “Move this particle exactly in this direction, at this speed.”<\/em><\/p>\n\n\n\n

Because Flow Matching uses ODEs, the trajectory from noise to data is deterministic<\/strong>, not random. More importantly, Flow Matching allows researchers to explicitly design<\/em> the probability path. Instead of being forced to follow the chaotic curves of a DDPM, engineers can define simple, direct flows. The neural network simply learns to match this optimal vector field.<\/p>\n\n\n\n

\"\"
Flow Matching vs Diffusion.<\/figcaption><\/figure>\n\n\n\n