Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence

• Main Authors: Tihanyi, Norbert, Bisztray, Tamas, Dubniczky, Richard A, Toth, Rebeka, Borsos, Bertalan, Cherif, Bilel, Ferrag, Mohamed Amine, Muzsai, Lajos, Jain, Ridhi, Marinelli, Ryan, Cordeiro, Lucas C, Debbah, Merouane
• Format: Journal Article
• Language: English
• Published: 20-10-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Multiagent Systems
• DOI: 10.48550/arxiv.2410.15490

As machine intelligence evolves, the need to test and compare the
problem-solving abilities of different AI models grows. However, current
benchmarks are often overly simplistic, allowing models to perform uniformly
well, making it difficult to distinguish their capabilities. Additionally,
benchmarks typically rely on static question-answer pairs, which models might
memorize or guess. To address these limitations, we introduce the Dynamic
Intelligence Assessment (DIA), a novel methodology for testing AI models using
dynamic question templates and improved metrics across multiple disciplines
such as mathematics, cryptography, cybersecurity, and computer science. The
accompanying DIA-Bench dataset, which includes 150 diverse and challenging task
templates with mutable parameters, is presented in various formats such as
text, PDFs, compiled binaries, and visual puzzles. Our framework introduces
four new metrics to assess a model's reliability and confidence across multiple
attempts. These metrics revealed that even simple questions are frequently
answered incorrectly when posed in varying forms, highlighting significant gaps
in models' reliability. Notably, models like GPT-4o tended to overestimate
their mathematical abilities, while ChatGPT-4o demonstrated better
decision-making and performance through effective tool usage. We evaluated
eight state-of-the-art large language models (LLMs) using DIA-Bench, showing
that current models struggle with complex tasks and often display unexpectedly
low confidence, even with simpler questions. The DIA framework sets a new
standard for assessing not only problem-solving but also a model's adaptive
intelligence and ability to assess its own limitations. The dataset is publicly
available on our project's website.