How can LLMs optimize multi-agent AI systems?
A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops
December 24, 2024
https://arxiv.org/pdf/2412.17149This paper introduces a framework for autonomously optimizing multi-agent AI systems using LLM-driven feedback loops. The system iteratively refines agent configurations (roles, tasks, workflows) based on qualitative and quantitative metrics to improve overall system performance without human intervention.
Key points for LLM-based multi-agent systems:
- LLM-driven refinement: An LLM (Llama 3.2-3B in this case) analyzes system outputs and generates hypotheses for improvement. These hypotheses inform changes to the agent system's configuration in subsequent iterations.
- Iterative feedback loop: The system operates in a continuous loop of execution, evaluation, hypothesis generation, and modification, driving iterative improvement towards pre-defined or dynamically generated and optionally human-revised criteria.
- Specialized agents: The framework employs specialized agents responsible for distinct phases of the refinement process (Refinement, Execution, Evaluation, Modification, and Documentation agents). This specialization improves efficiency and allows for more targeted refinements.
- Autonomous optimization: The entire process is designed to run autonomously, minimizing the need for manual intervention.
- Qualitative and quantitative metrics: The system uses both qualitative (clarity, relevance, etc.) and quantitative (execution time, success rate, etc.) metrics to evaluate system performance and guide the optimization process.