Masked Diffusion Models Get a Fair Evaluation: New Benchmark Exposes Training and Sampling Trade-offs

Masked diffusion models have emerged as a promising middle ground between autoregressive (AR) models and continuous diffusion models, promising tractable likelihoods and flexible sampling. But until now, a lack of standardized evaluation made it hard to compare them fairly against these alternatives. A new paper, "Masked Diffusion Models Are Secretly Zero-Shot Classifiers," aims to fix that with a rigorous benchmark that (spoiler) also uncovers some surprising training dynamics.

The authors introduce a unified evaluation framework that measures both likelihood (bits-per-dimension) and sample quality (FID/Inception Score) across multiple benchmark datasets, including CIFAR-10, ImageNet-32, and text-generation tasks. They also analyze the effect of training hyperparameters—specifically the number of denoising steps during training—and how different sampling schedules (e.g., uniform vs. cosine) impact performance.

Their key findings are nuanced. On likelihood, masked diffusion models are competitive with AR models and continuous diffusion, sometimes even matching them. But on sample quality, they consistently trail behind, particularly on high-resolution images. The authors hypothesize this is because masked diffusion's generative process is a discrete sequence of mask-and-predict steps, which may not capture long-range dependencies as effectively as continuous diffusion's denoising of the entire image at once.

Perhaps more counterintuitively, they find that training with more steps doesn't always improve sample quality. In fact, for certain datasets, training with fewer steps and then sampling with a different number of steps yields better FID scores. This suggests that the model's prior over mask schedules during training creates a mismatch with the sampling schedule, leading to suboptimal samples. The paper's title hints at another surprising discovery: masked diffusion models can be repurposed as zero-shot classifiers with impressive accuracy, rivaling dedicated classifiers on some benchmarks. This emerges because the model's learned mask-prediction objective naturally encodes class-conditional information.

For developers building generative models, this paper is a cautionary tale. It underscores that the choice of training and sampling schedules is not trivial and that simply porting ideas from continuous diffusion to the discrete masked setting can backfire. The authors provide concrete recommendations: use a cosine schedule for sampling, and avoid over-training with a large number of steps if your goal is sample quality.

The code is not yet public, but the evaluation framework is described in detail, making it possible to replicate and extend the work. As masked diffusion models gain traction in text and audio generation, having a solid empirical baseline is crucial. This paper delivers that, with enough nuance to keep practitioners on their toes.