Gradient Methods for Non‑convex Optimization

Abstract

Non-convex optimization forms bedrock of most modern
machine learning (ML) techniques such as deep learning. While nonconvex
optimization problems have been studied for the past several
decades, ML-based problems have significantly different characteristics
and requirements due to large datasets and high-dimensional parameter
spaces along with the statistical nature of the problem. Over the
last few years, there has been a flurry of activity in non-convex optimization
for such ML problems. This article surveys a few of the foundational
approaches in this domain.