Can LLMs automate social science content analysis?
SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Intervention
February 18, 2025
https://arxiv.org/pdf/2502.10937This paper introduces SCALE, a multi-agent framework using LLMs to automate the labor-intensive process of content analysis in social science. SCALE simulates human researchers by coding texts, engaging in discussions to resolve coding discrepancies, and refining the coding rulebook (codebook). Key points for LLM-based multi-agent systems include the use of distinct personas for each agent, integration of human feedback through collaborative and directive interventions, and the iterative evolution of the codebook based on agent discussions and expert input. Experiments show SCALE achieves human-level performance across various content analysis tasks.