How can LLMs learn social deduction via multi-agent RL?
Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning
February 11, 2025
https://arxiv.org/pdf/2502.06060This research explores how to train language models (LLMs) for effective communication and strategy in multi-agent social deduction games, specifically using a simplified version of Among Us.
Key points for LLM-based multi-agent systems:
- Emergent Communication: LLMs can learn complex communication strategies without human examples by using a combination of reinforcement learning (RL) and tailored reward signals.
- Listening and Speaking: The system separates training into "listening" (understanding messages and game state) and "speaking" (generating influential messages). Listening is trained via a supervised imposter prediction task, while speaking is trained through RL by rewarding messages that shift other agents' beliefs towards the correct imposter.
- Robustness: The trained LLM agents are robust against a variety of imposters and environment configurations.
- Self-Improvement: The framework allows LLMs to self-critique and improve their communication abilities over time without relying on human feedback.
- Natural Language Grounding: The game environment is designed to interact with agents using natural language, making it directly compatible with LLMs as agents.
- Challenges: Addressing issues like diverging from natural language, degenerate solutions, and the use of action tokens in discussions required specific training techniques.
- Ethical Considerations: Agents sometimes generate false statements for strategic advantage, raising potential ethical concerns for applications beyond the game environment.