Fine-grained Generalization Analysis of Structured Output Prediction

• Main Authors: Mustafa, Waleed, Lei, Yunwen, Ledent, Antoine, Kloft, Marius
• Format: Journal Article
• Language: English
• Published: 31-05-2021
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2106.00115

In machine learning we often encounter structured output prediction problems
(SOPPs), i.e. problems where the output space admits a rich internal structure.
Application domains where SOPPs naturally occur include natural language
processing, speech recognition, and computer vision. Typical SOPPs have an
extremely large label set, which grows exponentially as a function of the size
of the output. Existing generalization analysis implies generalization bounds
with at least a square-root dependency on the cardinality $d$ of the label set,
which can be vacuous in practice. In this paper, we significantly improve the
state of the art by developing novel high-probability bounds with a logarithmic
dependency on $d$. Moreover, we leverage the lens of algorithmic stability to
develop generalization bounds in expectation without any dependency on $d$. Our
results therefore build a solid theoretical foundation for learning in
large-scale SOPPs. Furthermore, we extend our results to learning with weakly
dependent data.