Contrastive Classification and Representation Learning with Probabilistic Interpretation

• Main Authors: Aljundi, Rahaf, Patel, Yash, Sulc, Milan, Olmeda, Daniel, Chumerin, Nikolay
• Format: Journal Article
• Language: English
• Published: 07-11-2022
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning
• DOI: 10.48550/arxiv.2211.03646

Cross entropy loss has served as the main objective function for
classification-based tasks. Widely deployed for learning neural network
classifiers, it shows both effectiveness and a probabilistic interpretation.
Recently, after the success of self supervised contrastive representation
learning methods, supervised contrastive methods have been proposed to learn
representations and have shown superior and more robust performance, compared
to solely training with cross entropy loss. However, cross entropy loss is
still needed to train the final classification layer. In this work, we
investigate the possibility of learning both the representation and the
classifier using one objective function that combines the robustness of
contrastive learning and the probabilistic interpretation of cross entropy
loss. First, we revisit a previously proposed contrastive-based objective
function that approximates cross entropy loss and present a simple extension to
learn the classifier jointly. Second, we propose a new version of the
supervised contrastive training that learns jointly the parameters of the
classifier and the backbone of the network. We empirically show that our
proposed objective functions show a significant improvement over the standard
cross entropy loss with more training stability and robustness in various
challenging settings.