Demystifying Orthogonal Monte Carlo and Beyond

• Main Authors: Lin, Han, Chen, Haoxian, Zhang, Tianyi, Laroche, Clement, Choromanski, Krzysztof
• Format: Journal Article
• Language: English
• Published: 27-05-2020
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2005.13590

Orthogonal Monte Carlo (OMC) is a very effective sampling algorithm imposing
structural geometric conditions (orthogonality) on samples for variance
reduction. Due to its simplicity and superior performance as compared to its
Quasi Monte Carlo counterparts, OMC is used in a wide spectrum of challenging
machine learning applications ranging from scalable kernel methods to
predictive recurrent neural networks, generative models and reinforcement
learning. However theoretical understanding of the method remains very limited.
In this paper we shed new light on the theoretical principles behind OMC,
applying theory of negatively dependent random variables to obtain several new
concentration results. We also propose a novel extensions of the method
leveraging number theory techniques and particle algorithms, called
Near-Orthogonal Monte Carlo (NOMC). We show that NOMC is the first algorithm
consistently outperforming OMC in applications ranging from kernel methods to
approximating distances in probabilistic metric spaces.