Deep Sketched Output Kernel Regression for Structured Prediction

• Main Authors: Ahmad, Tamim El, Yang, Junjie, Laforgue, Pierre, d'Alché-Buc, Florence
• Format: Journal Article
• Language: English
• Published: 13-06-2024
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2406.09253

By leveraging the kernel trick in the output space, kernel-induced losses
provide a principled way to define structured output prediction tasks for a
wide variety of output modalities. In particular, they have been successfully
used in the context of surrogate non-parametric regression, where the kernel
trick is typically exploited in the input space as well. However, when inputs
are images or texts, more expressive models such as deep neural networks seem
more suited than non-parametric methods. In this work, we tackle the question
of how to train neural networks to solve structured output prediction tasks,
while still benefiting from the versatility and relevance of kernel-induced
losses. We design a novel family of deep neural architectures, whose last layer
predicts in a data-dependent finite-dimensional subspace of the
infinite-dimensional output feature space deriving from the kernel-induced
loss. This subspace is chosen as the span of the eigenfunctions of a
randomly-approximated version of the empirical kernel covariance operator.
Interestingly, this approach unlocks the use of gradient descent algorithms
(and consequently of any neural architecture) for structured prediction.
Experiments on synthetic tasks as well as real-world supervised graph
prediction problems show the relevance of our method.