Can I control AI-generated driving scenarios?
Controllable Latent Diffusion for Traffic Simulation
This paper introduces Controllable Latent Diffusion (CLD), a new method for creating realistic and controllable traffic simulations for testing autonomous vehicles. It uses a diffusion model (DM) trained in a compressed latent space learned by a variational autoencoder (VAE). The key innovation is a reward-driven Markov Decision Process (MDP) that guides the DM's generation process, ensuring the simulated traffic follows rules and avoids collisions. This approach allows for generating diverse, realistic, and safe driving scenarios, addressing the limitations of existing methods.
For LLM-based multi-agent systems, CLD's combination of generative models (DMs) with reinforcement learning (reward-driven MDP) offers a compelling approach to generating realistic and controlled agent behaviors. The use of a latent space could also be beneficial for managing the complexity of LLM-based agent interactions, especially in environments with many agents or complex dynamics. This framework could potentially be adapted to train LLMs to generate multi-agent interaction sequences that adhere to specific rules or objectives.