Networks pose novel challenges for inference and learning due to their discrete, high-dimensional nature. This inherent complexity necessitates the development of statistically principled unsupervised learning objectives that steer clear of ad hoc heuristics to distinguish meaningful structure from noise in real networks. In this talk I will discuss a few recent projects aimed at developing principled unsupervised learning methods that parsimoniously summarize structural and dynamical regularities in network data of multiple forms: geographical networks, multilayer networks, temporal networks, and hypergraphs. These methods are unified under the Minimum Description Length principle from information theory, which readily permits fully nonparametric inference while explicitly highlighting particular regularities of interest in discrete datasets. I will discuss the motivation for this family of methods as well as a general procedure for applying this framework to other problems in network inference. I will also discuss its relationship with hierarchical Bayesian modeling, which allows for the comparison of parameter recovery performance across different optimization algorithms as well as further model selection with posterior predictive checking.