Probing Taxonomic and Thematic Embeddings for Taxonomic Information

• Main Authors: Klubička, Filip, Kelleher, John D
• Format: Journal Article
• Language: English
• Published: 25-01-2023
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Learning
• DOI: 10.48550/arxiv.2301.10656

Modelling taxonomic and thematic relatedness is important for building AI
with comprehensive natural language understanding. The goal of this paper is to
learn more about how taxonomic information is structurally encoded in
embeddings. To do this, we design a new hypernym-hyponym probing task and
perform a comparative probing study of taxonomic and thematic SGNS and GloVe
embeddings. Our experiments indicate that both types of embeddings encode some
taxonomic information, but the amount, as well as the geometric properties of
the encodings, are independently related to both the encoder architecture, as
well as the embedding training data. Specifically, we find that only taxonomic
embeddings carry taxonomic information in their norm, which is determined by
the underlying distribution in the data.