Marco Gherardi — Università degli Studi di Milano #
Universal training dynamics in deep learning #
While the recent success of deep learning in applications is staggering, our understanding of why and how this paradigm works so well is lagging behind. This divide calls for a multi-disciplinary effort, to which physicists are contributing on both the theoretical and the empirical side. A key concept, somewhat underrepresented in other disciplines but central to physics, is that of universality. I will present the empirical discovery of a potentially universal feature of training dynamics in deep neural networks [1]. The observed universal behavior boils down to the presence, robustly across different training sets and architectures, of special data points that are exceptionally conserved and particularly influential for the generalization capacity of the trained model.  [1] Ciceri et al., Nature Machine Intelligence 6, 40 (2024)