Can LLMs improve legal AI decision-making?
Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice
December 30, 2024
https://arxiv.org/pdf/2412.18697This paper proposes AgentsBench, a multi-agent framework using Large Language Models (LLMs) to simulate a judicial bench for more trustworthy legal decision-making. Agents representing judges and jurors deliberate, debate, and reach consensus, mirroring real-world judicial processes.
Key points for LLM-based multi-agent systems:
- Multi-Agent Deliberation: LLMs act as individual agents (judges and jurors) with distinct roles and perspectives, enabling a more nuanced and fair decision-making process compared to single LLM systems.
- Specialized Agent Roles: Agents are assigned roles with corresponding prompts influencing their behavior (e.g., judges moderate, jurors consider societal impact).
- Iterative Refinement: Agents engage in multiple rounds of deliberation, refining their initial sentencing proposals based on the arguments presented by other agents.
- Consensus-Based Decision Making: The system aims for consensus, reflecting real-world judicial practices. The presiding judge (an LLM agent) evaluates consensus dynamically.
- Enhanced Explainability & Trust: The framework captures individual reasoning and the overall deliberation process, offering increased transparency and trust compared to traditional LLM approaches. It also aims to improve the ethical dimension of decisions by incorporating diverse perspectives.
- Performance Gains: AgentsBench achieved higher accuracy in legal judgment prediction and especially scored better regarding morality considerations compared to baseline approaches using single LLMs.