Events

Physics ∩ ML Seminars

<< Fall 2021 Spring 2022 Summer 2022 >>
Subscribe your calendar or receive email announcements of events
BI for AI: Energy conserving descent for optimization
Date: Wednesday, March 16th
Time: 11:00 am
Place: Zoom link:
Speaker: Eva Silverstein, Stanford University
Abstract: We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks (so far at small scale), including generalization. It cannot stop at a high local minimum and cannot overshoot the global minimum, proceeds faster than GD+momentum in shallow valleys, and predictably finds multiple solutions according to a concrete formula for the measure on phase space which is applicable as a result of the energy conservation. Larger-scale experiments in progress are required to assess its relative performance on ML problems of current interest, along with further theoretical analysis its impact on representation/feature learning. Based on and ongoing work.
Add this event to your calendar