Bipedal humanoids have the potential to revolutionize work in human environments, but how far are we to realizing this vision? Despite growing interest, many challenges remain unresolved. In this talk, I will present our approach to addressing these challenges in bipedal robots by integrating optimal control and deep reinforcement learning (RL). Specifically, I will detail our development of a general RL framework that enables versatile, dynamic, and robust bipedal locomotion in the real world. Using the bipedal robot Cassie, this framework has demonstrated novel capabilities, including running a 400m dash, running over varied terrains and against perturbations, executing targeted standing jumps, and achieving compliant walking. While we will show that RL-based methods can unlock the robot’s full potential of agility, assessing the stability of such a system is an open question. To address this, I will discuss our efforts to bridge model-based safety-critical control and model-free RL by identifying low-dimensional models controlled by RL policies and applying stability and safety guarantees to these simplified models. Tackling these challenges not only paves the way for building reliable bipedal humanoids but also reshapes our view of the synergy between model-based and learning-based control methods.