Guest Speaker: Dr. Lin Xiao, Meta

Lecture: Quantization through Piecewise-Affine Regularization

Abstract: Quantization is a widely adopted approach for model compression, which can significantly reduce the memory, compute, and latency of modern AI models. However, current quantization methods are mostly based on heuristics and lack theoretical support. We propose a general framework of piecewise-affine regularization (PAR) for finding quantized solutions to continuous optimization problems and investigate its statistical properties and optimization methods. 

We show that in the over-parameterized regime of supervised learning, every critical point of the PAR-regularized loss function exhibits a high degree of quantization. For linear regression problems, we can approximate the classical formulations of ridge regression and Lasso using PAR and obtain similar statistical guarantees with quantized solutions. For optimization methods, we focus on the setting with a stochastic gradient oracle and discuss algorithms that can successfully identify the manifold of quantized solutions. 

Bio: Lin Xiao is a Research Scientist at Meta's Fundamental AI Research (FAIR) lab. He received a Bachelor of Engineering from Beijing University of Aeronautics and Astronautics (now Beihang University) and a PhD from Stanford University and was a postdoctoral fellow at California Institute of Technology. Before joining Meta (then Facebook) in 2020, he spent 14 years as a Researcher at Microsoft Research. He currently serves as an associate editor for the SIAM Journal on Optimization, Mathematical Programming, and the Journal of Machine Learning Research. His current research interests include theory and algorithms for large-scale optimization, reinforcement learning, and parallel and distributed computing.