Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models

• Main Authors: Colombo, Pierre, Pellegrain, Victor, Boudiaf, Malik, Storchan, Victor, Tami, Myriam, Ayed, Ismail Ben, Hudelot, Celine, Piantanida, Pablo
• Format: Journal Article
• Language: English
• Published: 21-10-2023
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2310.13998

Proprietary and closed APIs are becoming increasingly common to process
natural language, and are impacting the practical applications of natural
language processing, including few-shot classification. Few-shot classification
involves training a model to perform a new classification task with a handful
of labeled data. This paper presents three contributions. First, we introduce a
scenario where the embedding of a pre-trained model is served through a gated
API with compute-cost and data-privacy constraints. Second, we propose a
transductive inference, a learning paradigm that has been overlooked by the NLP
community. Transductive inference, unlike traditional inductive learning,
leverages the statistics of unlabeled data. We also introduce a new
parameter-free transductive regularizer based on the Fisher-Rao loss, which can
be used on top of the gated API embeddings. This method fully utilizes
unlabeled data, does not share any label with the third-party API provider and
could serve as a baseline for future research. Third, we propose an improved
experimental setting and compile a benchmark of eight datasets involving
multiclass classification in four different languages, with up to 151 classes.
We evaluate our methods using eight backbone models, along with an episodic
evaluation over 1,000 episodes, which demonstrate the superiority of
transductive inference over the standard inductive setting.