How can LLMs improve sim-to-real transfer for robots?
From Abstraction to Reality: DARPA's Vision for Robust Sim-to-Real Autonomy
This paper discusses the DARPA TIAMAT program, which aims to improve "sim-to-real" transfer for autonomous systems, especially in time-sensitive scenarios. The core idea is to train agents on diverse low-fidelity simulations augmented with shared semantic information (like knowledge graphs) instead of relying solely on computationally expensive high-fidelity simulations. This "abstract-to-real" approach seeks to foster faster adaptation to novel real-world situations.
Key points relevant to LLM-based multi-agent systems include leveraging semantic anchors (e.g., logic, natural language) for knowledge transfer and grounding representations, similar to how LLMs use syntax and semantics for few-shot learning. The program also highlights the need for robust decision-making under uncertainty and adapting internal model representations based on real-world feedback, mirroring refinement techniques used with LLMs. The diverse set of performers are exploring neuro-symbolic approaches, generative methods, and techniques inspired by foundation model training, showcasing the potential of these concepts for multi-agent systems.