Events

Physics ∩ ML Seminars

<< Spring 2020 Summer 2020 Fall 2020 >>
Subscribe your calendar or receive email announcements of events

Events on Wednesday, August 26th, 2020

The large learning rate phase of deep learning
Time: 11:00 am - 12:00 pm
Place: Please register for this online event:
Speaker: Yasaman Bahri, Google Brain
Abstract: Recent investigations of infinitely-wide deep neural networks have given rise to foundational connections between deep nets, kernels, and Gaussian processes. Nonetheless, there is still a gap to characterizing the dynamics of finite-width neural networks in common optimization settings. I’ll discuss how the choice of learning rate is a crucial factor to be considered and naturally classifies gradient descent dynamics of deep nets into two classes (a ‘lazy’ regime and a ‘catapult’ regime) which are separated by a sharp transition as networks become wider. I’ll discuss the distinct phenomenological signatures of the two phases and how they are elucidated in a class of solvable simple models we analyze.
Host: Shiu
Add this event to your calendar