How can LLMs handle real-time human-AI collaboration?
Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaboration
February 18, 2025
https://arxiv.org/pdf/2502.11882This paper introduces DPT-Agent, a framework for building AI agents that can collaborate with humans in real-time, like in a fast-paced video game. It addresses the challenge of LLMs being either too slow or too simple for effective real-time teamwork.
Key points for LLM-based multi-agent systems:
- Dual Process Theory (DPT): DPT-Agent combines a fast, rule-based system (System 1) with a slower, LLM-powered reasoning system (System 2) to balance speed and intelligence.
- Theory of Mind (ToM): System 2 uses ToM to infer the human player's intentions and adapt its strategy accordingly.
- Asynchronous Reflection: System 2 reflects on past performance to improve its strategy over time, without interrupting real-time actions.
- Code as Policy: System 2 generates code that controls System 1, allowing high-level reasoning to influence low-level actions.
- Overcooked Environment: The system is tested in a modified version of the Overcooked game, a challenging benchmark for multi-agent collaboration.
- Human Evaluation: Experiments show DPT-Agent performs better than existing frameworks when collaborating with both rule-based agents and real humans.