The rise of data-driven methods in robotics has significantly enhanced a robot’s capacity for perception, reasoning, and acting. However, the challenge and expense of collecting a diverse dataset with robots prevent learning control policies that are generalizable across various settings and tasks. Alternatively, while data sources like videos and robot play data are scalable, they are often not directly applicable due to the domain gaps and the absence of optimal action labels. In this talk, I will discuss my research on learning visual representations, particle trajectory models, and particle dynamics models from these data to learn generalizable low-level policies. These structured representations enables the learned policies to generalize to novel objects and configurations. I will conclude by demonstrating how these low-level skills can be assembled to tackle long-horizon and novel tasks.