HKU IDS Scholar Seminar Series #20:
Towards Multimodal and Interactive Visual Generation as World Models
Abstract
As generative models achieve increasingly greater performance, the research frontier is shifting toward new challenges of multimodal and interactive world models. This talk presents recent advancements and insights across three interconnected themes. First, we introduce unified frameworks for multimodal understanding and generation, exploring methods to enhance their semantic-spatial reasoning abilities. Second, we demonstrate interactive video generation systems that incorporate action control mechanisms, enabling gaming-like experiences where users dynamically influence content evolution. Our solutions address critical challenges in memory and 3D consistency during prolonged interaction sessions. Finally, we propose autoregressive visual generation architectures that inherently support multimodal integration and interactivity. Through systematic architectural innovations, we overcome longstanding bottlenecks in output quality and computational efficiency, establishing a viable alternative to diffusion-based paradigms. Looking into the future, we aim to build multimodal and interactive visual generation models as world models.
Speaker
Prof Xihui LIU
Assistant Professor @ HKU IDS & EEE Professor Xihui Liu is an Assistant Professor at the Department of Electrical and Electronic Engineering (EEE) and the Musketeers Foundation Institute of Data Science (IDS), The University of Hong Kong. Before joining HKU, she was a Postdoctoral Researcher at UC Berkeley working with Prof. Trevor Darrell. She received her Ph.D. degree from Multimedia Lab, The Chinese University of Hong Kong in 2021 and her Bachelor’s degree from Tsinghua University in 2017. She has won several awards such as Adobe Research Fellowship 2020, MIT EECS Rising Stars 2021, CVPR 2021 Doctoral Consortium Award, WAIC Rising Star Award 2022, CVPR Outstanding Reviewers Award, and ICLR Outstanding Reviewers Award. For full biography of Prof. LIU, please refer to: https://datascience.hku.hk/people/xihui-liu/
Moderator
Prof Ping LUO
Associate Professor @ HKU IDS & CS Professor Ping Luo’s researches aim at 1) developing Differentiable/ Meta/ Reinforcement Learning algorithms that endow machines and devices to solve complex tasks with larger autonomy, 2) understanding foundations of deep learning algorithms, and 3) enabling applications in Computer Vision and Artificial Intelligence. Professor Ping Luo received his PhD degree in 2014 in Information Engineering, the Chinese University of Hong Kong (CUHK), supervised by Prof. Xiaoou Tang (founder of SenseTime Group Ltd.) and Prof. Xiaogang Wang. He was a Research Director in SenseTime Research. He has published 70+ peer-reviewed articles (including 20 first author papers) in top-tier conferences and journals such as TPAMI, IJCV, ICML, ICLR, NeurIPS and CVPR. He has won a number of competitions and awards such as the first runner up in 2014 ImageNet ILSVRC Challenge, the first place in 2017 DAVIS Challenge on Video Object Segmentation, Gold medal in 2017 Youtube‐8M Video Classification Challenge, the first place in 2018 Drivable Area Segmentation Challenge for Autonomous Driving, 2011 HK PhD Fellow Award, and 2013 Microsoft Research Fellow Award (ten PhDs in Asia). For full biography of Prof. Ping LUO, please refer to: https://datascience.hku.hk/people/ping-luo/
Moderator
Prof. Yi Ma
Director; Professor, Chair of Artificial Intelligence @ HKU IDS & Department of Computer Science
Professor Yi Ma is a Chair Professor in the Musketeers Foundation Institute of Data Science (HKU IDS) and Department of Computer Science at the University of Hong Kong. He took up the Directorship of HKU IDS on January 12, 2023. He is also a Professor at the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. He has published about 60 journal papers, 120 conference papers, and three textbooks in computer vision, generalized principal component analysis, and high-dimensional data analysis.
Professor Ma’s research interests cover computer vision, high-dimensional data analysis, and intelligent systems. For full biography of Professor Ma, please refer to: https://datascience.hku.hk/people/yi-ma/
