HKU IDS Scholar Seminar Series #19:
Foundation Models as Embodied Agents: Towards AI That Talks to You and Acts on Your Behalf
Abstract
Recent advances in foundation models have enabled AI agents to operate across digital and physical environments through natural language interaction. In this talk, I will present our work on transforming large language models and vision-language models into embodied agents that execute real-world tasks on behalf of users. I will discuss three categories of agents: code agents that automatically generate code for data science and software development, computer-use agents that control your computer as humans do, and physical AI agents that take actions in the physical world. Our approach grounds natural language and perception into executable code and actions within their respective environments, enabling non-experts to access complex systems through conversational interfaces. I will cover our methods for transforming LLMs/VLMs into these agents, demonstrate how we evaluate their performance in embodied environments, and discuss the technical challenges and safety considerations of deploying AI agents that seamlessly bridge language understanding with actionable behaviors across digital and physical domains.
Speaker
Prof Tao YU
Assistant Professor @ HKU IDS & CDS Professor Tao Yu is an Assistant Professor in the Computer Science Department of the University of Hong Kong. He is also a Postdoctoral Research Fellow in the Department of Computer Science and Engineering at University of Washington and a co-director of the NLP group at the University of Hong Kong. His research interest is in Natural Language Processing and Deep Learning, with a focus on designing and building conversational natural language interfaces that can help humans explore and reason over data in any application (e.g., relational databases and mobile apps) in a robust and trusted manner. He has published and served in the program committee at ACL, EMNLP, ICLR, NAACL, etc. He co-organized the Interactive and Executable Semantic Parsing workshop at EMNLP 2020. For full biography of Prof. YU, please refer to: https://datascience.hku.hk/people/tao-yu/
Moderator
Dr Wenjie HUANG
Research Assistant Professor @ HKU IDS & DASE Dr. Wenjie Huang is Research Assistant Professor in Department of Data and Systems Engineering, The University of Hong Kong. He received Ph.D. degree from the Department of Industrial Systems Engineering and Management, National University of Singapore (NUS) in 2019 and B.S. degree in the Department of Industrial Engineering from Shanghai Jiao Tong University, China in 2014. Prior to joining HKU, he held joint postdoc positions at School of Data Science, The Chinese University of Hong Kong, Shenzhen and Group for Research in Decision Analysis (GERAD), Quebec, Canada. His research projects have been supported by NSFC research funds, NRF Singapore and NUS Young Investigator Award. For full biography of Dr. Huang, please refer to: https://datascience.hku.hk/people/wenjie-huang/
Moderator
Professor Yi Ma is a Chair Professor in the Musketeers Foundation Institute of Data Science (HKU IDS) and Department of Computer Science at the University of Hong Kong. He took up the Directorship of HKU IDS on January 12, 2023. He is also a Professor at the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. He has published about 60 journal papers, 120 conference papers, and three textbooks in computer vision, generalized principal component analysis, and high-dimensional data analysis.
Professor Ma’s research interests cover computer vision, high-dimensional data analysis, and intelligent systems. For full biography of Professor Ma, please refer to: https://datascience.hku.hk/people/yi-ma/
