HKU IDS Guest Seminar Series:
Pointwise Generalization in Deep Neural Networks
Abstract
We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, spectrum-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, spectrum-aware complexity.
Speaker
Prof Yunbei Xu
Assistant Professor, National University of Singapore Professor Yunbei Xu is a Presidential Young Assistant Professor at the National University of Singapore. He received a B.S. in Pure Mathematics from Peking University, a Ph.D. in Decision Sciences from Columbia Business School, and completed postdoctoral training in computing at MIT. He is a recipient of the ICML Outstanding Paper Award and First Place in the INFORMS Student Paper Competition. His research develops mathematical foundations for AI and real‑world systems, focusing on structural principles of neural networks and dynamical systems in language, decision-making, and the physical world. For full biography of Prof. Xu, please refer to: https://cde.nus.edu.sg/isem/staff/xuyunbei/
Moderator
Prof Yingyu Liang
Associate Professor @ HKU IDS & SCDS rofessor Yingyu Liang is an Associate Professor at the Musketeers Foundation Institute of Data Science and the Department of Computer Science at The University of Hong Kong, and at the Department of Computer Sciences at the University of Wisconsin–Madison. He received his Ph.D. from Georgia Tech, after degrees from Tsinghua University, and is a recipient of the NSF CAREER Award. His research focuses on theoretical foundations of modern machine learning, including optimization and generalization in deep learning and robust machine learning. For full biography of Prof. Liang, please refer to: https://datascience.hku.hk/people/yingyu-liang/
