The Legal Argument Reasoning Task in Civil Procedure

• Main Authors: Bongard, Leonard, Held, Lena, Habernal, Ivan
• Format: Journal Article
• Language: English
• Published: 05-11-2022
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2211.02950

We present a new NLP task and dataset from the domain of the U.S. civil
procedure. Each instance of the dataset consists of a general introduction to
the case, a particular question, and a possible solution argument, accompanied
by a detailed analysis of why the argument applies in that case. Since the
dataset is based on a book aimed at law students, we believe that it represents
a truly complex task for benchmarking modern legal language models. Our
baseline evaluation shows that fine-tuning a legal transformer provides some
advantage over random baseline models, but our analysis reveals that the actual
ability to infer legal arguments remains a challenging open research question.