How can I pick the best LLM agent for a task?
AgentRec: Agent Recommendation Using Sentence Embeddings Aligned to Human Feedback
January 24, 2025
https://arxiv.org/pdf/2501.13333This paper introduces AgentRec, a system for efficiently selecting the most appropriate Large Language Model (LLM) agent for a given task based on a natural language prompt. It uses sentence embeddings and a scoring mechanism based on cosine similarity and generalized p-means to recommend agents, achieving 92.2% accuracy on a synthetic dataset.
Key points for LLM-based multi-agent systems:
- Agent Selection: Addresses the challenge of dynamically choosing the right agent from a pool of specialized LLMs for a given user request.
- Sentence Embeddings: Leverages sentence embeddings to capture the semantic meaning of prompts and match them to the expertise of different agents.
- Reinforcement Learning from Human Feedback (RLHF): Uses RLHF to align the agent recommendation system with human expectations and values.
- Performance: Demonstrates fast agent selection (under 300ms) suitable for real-time applications.
- Synthetic Data Generation: Introduces a method for generating synthetic data to train the system, addressing the scarcity of real-world multi-agent datasets.
- Open Source: Code and data are publicly available for experimentation and further development.