In the field of language models, approaches using continuous diffusion models are once again attracting attention. While the field experienced stagnation for several years, an increase in recent research indicates that this method is resurfacing.
Approaches to Continuous Diffusion Models in Language Models Regain Attention
This article is a translation. Read the Japanese original
Current mainstream autoregressive models function by generating tokens sequentially, one by one. In contrast, diffusion models perform generation by reversing a process of information destruction.
In the past, discrete diffusion methods, which are suitable for categorical data, were the mainstream. However, since 2022, methods have emerged that represent text as continuous embedding vectors and apply a corruption process using Gaussian noise.
This approach, represented by Diffusion-LM, is said to have advantages, particularly in controllable text generation. Following this, derivative methods such as DiffuSeq and SSD-LM have also been researched.
Source: Continuous Diffusion Language Models (CDLM's) (Hacker News Frontpage, 2026-08-31)