Physics ∩ ML Seminars
Events on Wednesday, March 16th, 2022
- BI for AI: Energy conserving descent for optimization
- Time: 11:00 am - 12:15 pm
- Place: Zoom link:
- Speaker: Eva Silverstein, Stanford University
- Abstract: We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks (so far at small scale), including generalization. It cannot stop at a high local minimum and cannot overshoot the global minimum, proceeds faster than GD+momentum in shallow valleys, and predictably finds multiple solutions according to a concrete formula for the measure on phase space which is applicable as a result of the energy conservation. Larger-scale experiments in progress are required to assess its relative performance on ML problems of current interest, along with further theoretical analysis its impact on representation/feature learning. Based on and ongoing work.