VAMO: Efficient Large-Scale Nonconvex Optimization via Adaptive Zeroth Order Variance Reduction

Abstract

We introduce VAMO, a variance-reduced mixed-gradient optimizer that combines full-batch zeroth-order gradients with mini-batch stochastic updates. Its hybrid estimator achieves a dimension-agnostic convergence rate while using less dynamic memory than many first-order baselines. Neural-network training and LLM fine-tuning experiments show strong performance with a smaller memory footprint.