Can LLMs automate research code generation?
Research CodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies
-
This paper introduces ResearchCodeAgent, a multi-agent system that uses LLMs to automatically translate research methodologies described in machine learning papers into working code. Given a research paper, relevant datasets, and starter code, the system aims to automate the often tedious process of implementing the described methodology for benchmarking or building upon.
-
ResearchCodeAgent employs a flexible multi-agent architecture with a defined action space allowing LLM agents to interact with a research environment (methodology description, data, code). It uses a dynamic planning mechanism with short and long-term memory, adapting its approach iteratively. This iterative refinement, along with programmatic safeguards against common LLM issues like looping, distinguishes it from single LLM calls or prescribed action sequences. Evaluation across diverse ML tasks shows ResearchCodeAgent generates higher quality, less error-prone code and offers significant time savings, particularly for complex tasks. This suggests the potential of multi-agent LLM systems for automating and accelerating research implementation workflows.