Smoothness-Adaptive Sharpness-Aware Minimization for Finding Flatter Minima

Abstract

The sharpness-aware minimization (SAM) procedure recently gained increasing attention due to its favorable generalization ability to unseen data. SAM aims to find flatter (local) minima, utilizing a minimax objective. An immediate challenge in the application of SAM is the adjustment of two pivotal step sizes, which significantly influence its effectiveness. We introduce a novel, straightforward approach for adjusting step sizes that adapts to the smoothness of the objective function, thereby reducing the necessity for manual tuning. This method, termed Smoothness-Adaptive SAM (SA-SAM), not only simplifies the optimization process but also promotes the method’s inherent tendency to converge towards flatter minima, enhancing performance in specific models.

Publication
Practical Machine Learning for Low Resource Settings Workshop, ICLR 2024
comments powered by Disqus