Publications

Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning, yet a fundamental practical question remains unresolved: how should the adapter rank be chosen? An overly small rank may lead to a poorly conditioned optimization landscape, whereas an unnecessarily large rank sacrifices the efficiency that motivates LoRA in the first place. Existing theoretical analyses provide only limited guidance on this trade-off, and their guarantees are typically established under restrictive theoretical settings. We address this gap by developing a substantially sharper landscape theory for LoRA, building on modern results from nonconvex low-rank matrix sensing. Our central insight is that the appropriate adapter rank should depend on the quality of the data-induced optimization geometry, rather than on the model alone. To formalize this connection, we introduce LoRA-RIP, a data-dependent restricted-isometry metric that characterizes the conditioning of the cross-entropy (CE) objective along LoRA-relevant low-rank directions. We prove that sufficient rank over-parameterization, with the required rank explicitly determined by the LoRA-RIP constant, eliminates spurious local minima, thereby extending existing RIP-based guarantees beyond the classical 1/3 regime. This characterization further enables principled data selection under a fixed rank budget. Experiments across language and vision tasks support these theoretical predictions, showing that rank and data quality are two coupled resources that should be jointly considered for more efficient and reliable LoRA fine-tuning.

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide a simple way to improve the pretraining-to-quantization transition. But the mechanism behind weight averaging remains insufficiently explained. This leads to inconsistent and fragile performance gains, thereby preventing practitioners from applying such a technique confidently. As a response, we formulate weight averaging as a trade-off between retaining training progress and improving robustness under perturbation. We further derive a continuous family of averaging kernels that unifies conventional strategies and achieves the Pareto frontier between the two competing goals. Critically, a theoretical framework for performing weight averaging under PTQ is developed. It can be shown that coarser quantization is more susceptible to perturbations, whereas finer quantization could be less affected. Thus, our results could provide unified theoretical guidance for performing weight averaging under different PTQ conditions. Experiments validate both the predicted behavior and the proposed averaging strategy. Code is available at https://github.com/MOFA-LAB/weight-averaging-for-ptq.

Recent work incorporates reusable skills distilled from past interactions into multimodal agent training, providing procedural guidance for long-horizon planning and tool use. However, policy optimization in these methods remains driven primarily by sparse outcome rewards, providing little supervision for intermediate decisions. Rubric-based rewards address this limitation through explicit intermediate criteria, but reliable rubrics are difficult to construct at scale and often disconnected from the procedure followed by the actor. We observe that a well-structured skill naturally specifies both how to act and what successful execution should achieve. Based on this insight, we introduce SkillRubric, which represents each skill through aligned actor-facing guidance and an evaluator-facing rubric. A multimodal verifier evaluates skill-defined goals using screenshots and tool outputs, assigning completion and progress rewards to the responsible turns. We further introduce an alternating co-evolution scheme that validates guidance revisions through paired rollouts under a frozen policy and rubric revisions offline under fixed guidance. Experiments across diverse multimodal agent benchmarks demonstrate consistent performance gains, while controlled paired rollouts further show that evolved skills provide more effective guidance for planning and tool use than their preceding versions.

Muon has recently emerged as a promising alternative to AdamW for language model pretraining by orthogonalizing momentum matrices using Newton–Schulz iterations. Although Muon mitigates gradient anisotropy, it does not explicitly account for the curvature geometry of the loss landscape and may therefore remain sensitive to curvature anisotropy. We bridge this gap by proposing MALT (Muon Augmented by Lightweight Two-sided Preconditioning), which uses lightweight diagonal preconditioners to reduce the sensitivity of Muon to curvature anisotropy. Specifically, MALT uses two-sided diagonal preconditioners with low memory and computational overhead to approximately capture the curvature geometry of the loss landscape. It orthogonalizes the preconditioned momentum using Newton–Schulz iterations and maps the result back to define the update direction, while norm grafting controls the update magnitude. To improve the robustness of MALT to stochastic gradient noise, we further propose MALTER (MALT with Adaptive stEpsize Rescaling). Convergence guarantees are provided for MALT in the stochastic non-convex setting. Experiments on GPT-2 Small, Medium, and Large pretraining show that the proposed methods outperform Muon while maintaining nearly the same memory footprint and wall-clock time.

Recently, Muon has gained substantial attention as an appealing alternative to Adam-like optimizers, with many works highlighting its advantages through spectral normalization and improved conditioning. Yet this positive theoretical narrative contrasts with its empirical performance in large language model (LLM) training, where Muon’s gains over Adam/AdamW are often mixed, schedule-sensitive, and not uniformly superior. To address this gap, we develop a trajectory-level theory characterizing both the strengths and limitations of Muon. We introduce a mixed-spiked matrix sensing model whose sensing operator decomposes into signal, spike, and bulk components, capturing a mixture of anisotropic structure and long-tail information reminiscent of LLM training. On top of it, we adopt a river-valley perspective in which we view the landscape as composed of a river direction flowing to the desired solution and hill directions encoding nuisance or task-irrelevant information. In the momentum-free setting, we show that Muon moves faster along the information-bearing river direction during early optimization, but can converge much more slowly near the river bottom than gradient descent. We then extend the river-valley perspective to general nonconvex objectives with momentum by studying points on the spectral river. There, while Muon converges faster early on, its orthogonalized update removes residual scale information, making it prone to overshooting and oscillation near the target solution. Together, these results suggest that our characterizations extend beyond spiked matrix sensing and motivate switching to GD-like refinement optimizers in the final phase, rather than relying only on a fixed learning-rate schedule for Muon. We also provide preliminary evidence supporting this two-stage approach in language model training experiments.

Zeroth-order (ZO) optimization has become increasingly popular and important in fine-tuning large language models (LLMs), especially on edge devices due to its ability to adjust the model to local data without the need for memory-intensive back-propagation. Recent works try to reduce ZO variance through low-dimensional subspace search, but subspace restriction alone leaves key optimization geometry under-exploited, motivating additional acceleration. In this work, we focus on the hidden layer training problem in which spectral optimizers like Muon outperform AdamW due to its ability to exploit weak spectral directions by orthogonalization. However, we have discovered that unlike in the first-order setting, full orthogonalization works poorly in the ZO setting since the gradient estimates are highly noisy and unreliable. To address this issue, we propose applying partial spectral orthogonalization to accelerate ZO optimization. To do so, we replace the iconic Newton–Schulz procedure in Muon with the faster, more concentrated power-iteration method so that it only amplifies dominant spectral directions. Furthermore, to improve the efficiency and generalization of the algorithm, we adopt a streaming variant of power-iteration that requires low variance in gradients, which is achieved through constraining our search inside a subspace obtained through the projection of momentum, echoing recent advances. Experiments on LLM fine-tuning show that our method can achieve from 1.5x to 4x the convergence speed of ZO-Muon, the current SOTA algorithm, across SuperGlue datasets in the OPT-13B model. Across different models, we also reach competitive final accuracies with less time in most cases compared with strong ZO baselines such as MeZO, LOZO and ZO-Muon. Code is available at https://github.com/MOFA-LAB/ZO-MOPI.git.

We propose a deterministic simulated-oracle-direction mechanism for escaping spurious local minima in low-rank matrix sensing without explicitly lifting the problem. The method projects escape directions from an overparametrized landscape back into the original parameter space and guarantees objective decrease from existing local minima. Numerical experiments show reliable escapes with low computational overhead.

We study how loss-function choice affects robustness and optimization landscapes in noisy matrix sensing. A kernel-based residual loss matches mean-squared error under Gaussian noise while remaining stable for heavy-tailed and heterogeneous corruption. Theoretical and empirical results show improved robustness and convergence across diverse noise distributions.

We study matrix sensing through the convex matrix-lasso relaxation, where the nuclear norm replaces rank. We show that the relaxation can certifiably recover a low-rank ground-truth matrix and establish a connection to its Burer–Monteiro factorization. The guarantees extend to finite-variance noise and clarify the relationship between convex and non-convex formulations.

We introduce VAMO, a variance-reduced mixed-gradient optimizer that combines full-batch zeroth-order gradients with mini-batch stochastic updates. Its hybrid estimator achieves a dimension-agnostic convergence rate while using less dynamic memory than many first-order baselines. Neural-network training and LLM fine-tuning experiments show strong performance with a smaller memory footprint.

We study certifying ReLU neural networks against adversarial input perturbations by partitioning the input uncertainty set and solving relaxations on each part. We show how intelligently designed branching can eliminate relaxation error for linear programs and reduce worst-case error for semidefinite programs. Experiments on MNIST, CIFAR-10, and Wisconsin breast-cancer classifiers demonstrate substantially higher certification rates.

We introduce Topology-Aware 3D Gaussian Splatting, combining local persistent Voronoi interpolation with a persistent-homology regularizer. The method improves geometric coverage and preserves topological structure during optimization. Experiments across novel-view-synthesis benchmarks improve PSNR, SSIM, and LPIPS while maintaining efficient memory use.

We analyze matrix-sensing landscapes far from the ground truth, where spurious solutions can obstruct local optimization. Under suitable conditions, distant critical points are strict saddles rather than local minima, and higher-order losses amplify the negative curvature around them. This provides a theoretical alternative to increasing model complexity through overparametrization.

We examine how gradient descent induces implicit regularization in tensor optimization through a lifted matrix-sensing framework. With sufficiently small initialization, gradient descent produces approximately rank-one tensors and critical points with escape directions. The results highlight how tensor parametrization and first-order methods can work together to reach global optima.

We study over-parametrization for low-rank matrix sensing by constructing an infinite hierarchy of lifted non-convex problems through tensor lifting and Burer–Monteiro factorization. Spurious solutions remain stationary through the hierarchy but become strict saddle points under technical conditions, allowing local search methods to escape them. We also derive a bound on how much over-parametrization is needed to eliminate spurious solutions.

We compare semidefinite programming and Burer–Monteiro factorization for matrix sensing. We identify structured instances where one method succeeds while the other fails, and show that the success of semidefinite programming depends on the rank of the unknown solution whereas Burer–Monteiro performance is rank-independent. The results clarify the distinct strengths of the two approaches.

We develop a mathematical framework for low-rank matrix optimization with noisy data under a less restrictive restricted-isometry condition than prior work. When the noiseless RIP constant is below one third, spurious local solutions of the noisy problem must remain close to the ground truth, while the strict-saddle analysis gives polynomial-time search guarantees. The results characterize local-minimum geometry across both favorable and difficult regimes.

We study low-rank matrix recovery with noisy linear measurements and ask when local search can find the ground truth with small error. We derive sharp guarantees for local minimizers under a restricted-isometry assumption, show how the guarantee improves as noise decreases, and prove a strict-saddle property that supports polynomial-time convergence of perturbed gradient descent.

We study noisy matrix sensing in both exact-parametrization and overparametrization regimes. The analysis quantifies how the restricted-isometry property shapes the non-convex landscape, bounds the distance of local minimizers from the ground truth, and proves a strict-saddle property that yields global convergence of perturbed gradient descent.

We develop a sequential semidefinite-programming framework for neural-network robustness certification. By adding non-convex cuts through disjunctive programming, the method shrinks the relaxation gap toward zero. We analyze the approach theoretically and empirically as the number of cuts increases.

We study certification of neural-network robustness to perturbed and adversarial inputs. A partition-based procedure exploits the structure of ReLU networks to tighten existing linear-programming relaxations, reaching zero relaxation error with a finite partition. Experiments show that the method can issue robustness certificates where prior approaches fail.

We present a certifiably globally optimal algorithm for determining the extrinsic calibration between two sensors that can produce independent egomotion estimates. The problem is formulated as a quadratically constrained quadratic program and solved through a semidefinite-programming relaxation with a global optimality certificate. The method is fast, robust to noise, and does not require a prior estimate or operator intervention.