Skip to content

HKU IDS Scholar Seminar Series #12: Achilles' Heel in Manipulation: Key Recipe and Missing Pieces towards Intelligent Embodied AI

Title: Achilles’ Heel in Manipulation: Key Recipe and Missing Pieces towards Intelligent Embodied AI
Speaker: Professor Hongyang Li, Assistant Professor, HKU IDS 
Date: November 28, 2024
Time: 10:30am – 11:30am

Venue: IDS Seminar Room, P603, Graduate House / Zoom 
Mode: Hybrid. Seats for on-site participants are limited. A confirmation email will be sent to participants who have successfully registered.

Abstract

The increasing demand for versatile robotic systems to operate in diverse and dynamic environments has emphasized the importance of a generalist policy, which leverages a large cross-embodiment data corpus to facilitate broad adaptability and high-level reasoning. However, the generalist would struggle with inefficient inference and cost-expensive training. The specialist policy, instead, is curated for specific domain data and excels at task-level precision with efficiency. Yet, it lacks the generalization capacity for a wide range of applications. Inspired by these observations, we introduce RoboDual, a synergistic dual-system that supplements the merits of both generalist and specialist policy. A diffusion transformer-based specialist is devised for multi-step action rollouts, exquisitely conditioned on the high-level task understanding and discretized action output of a vision-language-action (VLA) based generalist. Compared to OpenVLA, RoboDual achieves a 12% improvement on CALVIN and 26.7% in real-world by adapting the specialist policy with 20M trainable parameters only. It maintains strong performance with merely 5% of demonstration data, and enables a 3.8 higher control frequency in real-world deployment. Code and models would be made publicly available.

Speaker

Prof. Hongyang LI
Assistant Professor @ HKU IDS
Professor Li is an Assistant Professor in HKU Musketeers Foundation Institute of Data Science and Research Scientist at OpenDriveLab, Shanghai AI Lab. His research focus is on autonomous driving and embodied AI. He led the end-to-end autonomous driving project, UniAD and won the IEEE CVPR 2023 Best Paper Award. UniAD has a large impact both in academia and industry, including the recent rollout to customers by Tesla in FSD V12. He proposed the bird’s-eye-view perception work, BEVFormer, that won Top 100 AI Papers in 2022 and was explicitly recognized by Jensen Huang, CEO of NVIDIA and Prof. Shashua, CEO of Mobileye at public keynotes. He served as Area Chair for CVPR 2023, 2024, NeurIPS 2023 (Notable AC), 2024, ACM MM 2024, ICLR 2025, referee for Nature Communications. He will serve as Workshop Chair for CVPR 2026. He is the Working Group Chair for IEEE Standards under Vehicular Technology Society and Senior Member of IEEE.
For full biography of Prof. LI, please refer to: https://datascience.hku.hk/people/hongyang-li/

Moderator

Prof. Yi Ma
Director; Professor, Chair of Artificial Intelligence @ HKU IDS & Department of Computer Science 

Professor Yi Ma is a Chair Professor in the Musketeers Foundation Institute of Data Science (HKU IDS) and Department of Computer Science at the University of Hong Kong. He took up the Directorship of HKU IDS on January 12, 2023. He is also a Professor at the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. He has published about 60 journal papers, 120 conference papers, and three textbooks in computer vision, generalized principal component analysis, and high-dimensional data analysis. 

Professor Ma’s research interests cover computer vision, high-dimensional data analysis, and intelligent systems. For full biography of Professor Ma, please refer to: https://datascience.hku.hk/people/yi-ma/

For information, please contact:
Email: datascience@hku.hk