A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence...

Full description

Saved in:
Bibliographic Details
Main Authors: Fornaciari, Francesca De Luca, Altuna, Begoña, Gonzalez-Dios, Itziar, Melero, Maite
Format: Journal Article
Language:English
Published: 17-05-2024
Subjects:
Computer Science - Computation and Language
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.
DOI:10.48550/arxiv.2405.10579