Diffusion models (DMs) excel in generative modeling, but their theoretical foundations and limitations remain underexplored. This talk addresses two key aspects: their feature learning dynamics and their ability to capture hidden inter-feature rules. First, I show that the denoising objective encourages DMs to learn balanced and comprehensive data representations, unlike classification models that prioritize easy-to-learn patterns. Theoretical analysis and experiments on synthetic and real-world datasets highlight this distinction. Next, I explore a critical limitation: DMs often fail to learn fine-grained hidden rules between dependent features, such as the relationship between the height of the sun and shadow length in images. Empirical evaluations on models like Stable Diffusion reveal consistent failures, supported by synthetic tasks and theoretical insights showing that denoising score matching (DSM) is incompatible with enforcing rule conformity. I discuss potential solutions, such as classifier-guided sampling, and their limitations. This talk provides a deeper understanding of DMs’ strengths and weaknesses, offering insights for building more robust and interpretable generative models.