The rise of data-driven methods in robotics has significantly enhanced a robot’s capacity for perception, reasoning, and acting. However, the challenge and expense of collecting a diverse dataset with robots prevent learning control policies that are generalizable across various settings and tasks. Alternatively, while data sources like videos and robot play data are scalable, they are often not directly applicable due to the domain gaps and the absence of optimal action labels. In this talk, I will discuss my research on learning visual representations, particle trajectory models, and particle dynamics models from these data to learn generalizable low-level policies. These structured representations enables the learned policies to generalize to novel objects and configurations. I will conclude by demonstrating how these low-level skills can be assembled to tackle long-horizon and novel tasks.

Videos are a beautiful way to share our lives, ideas, stories, and emotions. Recent generative models (e.g. SORA) can generate photorealistic short videos. However, they remain far from being able to create complicated artwork (e.g. films) that requires more human creativity due to the huge gap between human mental representations on control signals and computer representations on pixels, timestamps. To address this challenge, I design controllable and reliable tools to bridge this gap such that creators interact with the tool with conceptual-friendly control signals and produce desired content in a more efficient way. Each module in the tool is reliable and explainable, which allows the creators to input their intentions, get their expected outputs, and know what happened within it. This allows creators to make iterative improvements in the video creation process rather than numerous trial-and-errors.

A majority of existing research in large language models and generative AI systems focuses on scaling and engineering. In this talk, I argue that we need a principled understanding of the science of generative AI, in particular, to understand the reasoning ability of large language models. First, I present a Bayesian latent variable approach to enhancing in-context learning in large language models (LLMs) through optimal demonstration selection, demonstrating substantial improvements across various text classification tasks. Second, I argue that modern generative AI systems must be modular and collaborative to solve complex reasoning problems. I will introduce Logic-LM, a locally grounded neuro-symbolic framework that synergizes LLMs with symbolic solvers, significantly boosting logical problem-solving abilities. We will also briefly elaborate on how to build neuro-symbolic solutions to improve the compositionality in text-to-image systems. Our observations indicate that the future of generative AI is modular and collaborative, as opposed to a single-model system.