Learning Non-Parametric Invariances from Data with Permanent Random Connectomes

• Main Authors: Pal, Dipan K, Chawla, Akshay, Savvides, Marios
• Format: Journal Article
• Language: English
• Published: 12-11-2019
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning; Computer Science - Neural and Evolutionary Computing; Statistics - Machine Learning
• DOI: 10.48550/arxiv.1911.05266

One of the fundamental problems in supervised classification and in machine
learning in general, is the modelling of non-parametric invariances that exist
in data. Most prior art has focused on enforcing priors in the form of
invariances to parametric nuisance transformations that are expected to be
present in data. Learning non-parametric invariances directly from data remains
an important open problem. In this paper, we introduce a new architectural
layer for convolutional networks which is capable of learning general
invariances from data itself. This layer can learn invariance to non-parametric
transformations and interestingly, motivates and incorporates permanent random
connectomes, thereby being called Permanent Random Connectome Non-Parametric
Transformation Networks (PRC-NPTN). PRC-NPTN networks are initialized with
random connections (not just weights) which are a small subset of the
connections in a fully connected convolution layer. Importantly, these
connections in PRC-NPTNs once initialized remain permanent throughout training
and testing. Permanent random connectomes make these architectures loosely more
biologically plausible than many other mainstream network architectures which
require highly ordered structures. We motivate randomly initialized connections
as a simple method to learn invariance from data itself while invoking
invariance towards multiple nuisance transformations simultaneously. We find
that these randomly initialized permanent connections have positive effects on
generalization, outperform much larger ConvNet baselines and the recently
proposed Non-Parametric Transformation Network (NPTN) on benchmarks that
enforce learning invariances from the data itself.