Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing

Abstract

We examine how gradient descent induces implicit regularization in tensor optimization through a lifted matrix-sensing framework. With sufficiently small initialization, gradient descent produces approximately rank-one tensors and critical points with escape directions. The results highlight how tensor parametrization and first-order methods can work together to reach global optima.