When is multi-agent orchestration worthwhile?
When Should We Orchestrate Multiple Agents?
March 19, 2025
https://arxiv.org/pdf/2503.13577This paper explores how to best utilize multiple agents (e.g., humans, LLMs, human+LLM) for a given task, considering factors like agent cost, availability, and expertise in different sub-tasks ("regions"). It introduces a framework for dynamically selecting the most appropriate agent for each sub-task as data arrives, rather than pre-assigning agents.
For LLM-based multi-agent systems, the key points are:
- Dynamic Orchestration: Real-time agent selection based on context and estimated utility can outperform static assignments and improve overall accuracy.
- Cost and Constraints: Incorporating real-world factors like LLM API costs and regulatory limitations into the orchestration framework significantly influences optimal agent selection.
- Human-in-the-Loop: Humans can be valuable agents within a multi-agent system, and the framework addresses how to balance human and AI contributions.
- User Studies: Experiments demonstrate that users struggle to effectively leverage multiple agents without orchestration, suggesting the value of automated assistance for agent selection in real-world applications.
- Addressing Rogers' Paradox: The framework can be used to mitigate the negative impacts of naive social learning in multi-agent scenarios where agents learn from each other, showing a path to effective human-AI collaboration in complex environments.