Can LLMs creatively deceive in Balderdash?
Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash
November 18, 2024
https://arxiv.org/pdf/2411.10422This paper explores the creativity and logical reasoning of Large Language Models (LLMs) using a simulated multi-agent version of the game Balderdash. Players (LLMs) try to create convincing fake definitions for obscure words to fool other players while also identifying the correct definition.
Key points for LLM-based multi-agent systems:
- LLMs can be used as agents in a game environment with a centralized game engine and communicate through defined prompts.
- Performance was evaluated using metrics like generating true definitions, creating deceptive definitions, and correctly guessing definitions.
- Providing LLMs with game history improves performance on common words but not on infrequent words, suggesting a limitation in reasoning with unfamiliar vocabulary.
- LLMs struggle to consistently choose the optimal strategy even when it's clearly advantageous based on game rules. This highlights a challenge in reasoning about rules and long-term strategy.
- A dedicated LLM was used as a "judge" to evaluate the semantic equivalence of definitions. This introduces potential bias, and alternative judging mechanisms could be explored.