How can I automate complex LLM workflow tuning?
Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning
This paper introduces Cognify, a framework for automatically optimizing complex workflows involving multiple Generative AI models (Gen-AI workflows), tools, and data retrieval. Manual tuning of such workflows is time-consuming and inefficient. Cognify uses a novel hierarchical search algorithm, AdaSeek, to efficiently explore the space of possible workflow improvements, considering changes to workflow structure, individual steps (like model selection or code changes), and edge weights (like prompt engineering). AdaSeek adapts its search based on the available budget and observed results, making it particularly relevant to LLM-based multi-agent systems where evaluation can be costly. Key points relevant to LLM-based multi-agent systems include the use of LLMs for proposing and evaluating workflow changes, the focus on non-differentiable components and discrete metrics common in these systems, and the ability to optimize under budget constraints.