Elias Stengel-Eskin

Major Update: I will be joining UT Austin as an Assistant Professor in the Department of Computer Science in Fall 2025! I will be recruiting Ph.D. students for Fall 2026 and also looking for interns throughout 2025-2026, please see this page for more details.

I am a Postdoctoral Research Associate at the University of North Carolina, Chapel Hill in the MURGe-Lab led by Mohit Bansal. I received my Ph.D. in 2023 from Johns Hopkins University, where I was supervised by Benjamin Van Durme and supported by an NSF GRFP.

I aim to develop AI agents that can intelligently communicate and collaborate with people and each other. My work addresses three key problems:

1. A central focus of my work is multi-agent communication and collaboration, which has led to work on multi-LLM multi-round discussions/debates, distilling multi-agent behavior, pragmatic/verbalized uncertainty, and persuasion.
2. Agents must be grounded to the world through their inputs and actions: another line of my work covers multimodal grounding and converting language to action through semantic parsing, text-to-code, and learning abstractions and skills.
3. Developing safe and robust agents means handling uncertainty, ambiguity, and underspecification. As we scale up tasks, underspecification and ambiguity will become increasingly relevant, especially when predicting actions/grounding to the world. My work covers calibration and uncertainty especially in connection with implicit phenomena such as vagueness, underspecification, and ambiguity. While I’ve mostly explored these topics through a linguistic lens, I am interested in their importance to intelligence more broadly.

Concretely, some of the main areas I’ve been publishing on recently are:

• Confidence Estimation and Calibration:
  • on multi-agent pragmatic reasoning for better calibration (Stengel-Eskin et al., NeurIPS 2024)
  • on calibration in semantic parsing (Stengel-Eskin and Van Durme, TACL 2023)
  • on balancing safety and usability using confidence (Stengel-Eskin and Van Durme, EMNLP 2023)
• Ambiguity and Underspecification:
  • on ambiguity in semantic parsing (Stengel-Eskin and Van Durme, ICLR 2024)
  • on improving VQA through preemptive self-clarification (Prasad et al., ICLR 2024)
  • on rephrasing and analyzing ambiguous questions in VQA (Stengel-Eskin et al. ACL 2023)
• Multi-Agent/Multi-Model Reasoning:
  • on multi-agent training to accept good and resist bad persuasion (Stengel-Eskin et al., NAACL 2025)
  • on structured distillation to learn from multi-agent discussions (Chen et al., ICML 2024)
  • on a new benchmark to assess game-theoretic abilities for LLM agents (Duan et al., NeurIPS 2024)
  • on multi-agent iterative coarse-to-fine refinement for reasoning tasks (Chen et al., 2024)
  • on using bandits to select instance-level reward models for LLM alignment (Nguyen et al., 2024)
  • on combining 16 LLMs on 1 GPU via symbolic MoE (Chen et al., 2025)
• Learning Skills and Abstractions for Agents/Coding/Planning:
  • on learning coding abstractions with LLMs (Stengel-Eskin et al., ICML 2024)
  • on discovering skills for RL agents (Fu et al., ICML 2024)
  • on distilling skills from teachers for LLM agents (Hashemzadeh et al., CoLLAs 2024)
  • on hybrid and controllable System 1 and System 2 planning (Saha et al., ICLR 2025)
  • on learning to generate unit tests for LLM debugging (Prasad et al., 2025)
  • on inferring structured abstractions for reasoning (Khan et al., 2025)
• Improving Multimodal Models and LLM Agents:
  • on building and testing data generation agents for creating training data (Khan et al., ICLR 2025 Spotlight)
  • on a tree-based representation for LLM-based video reasoning (Wang et al. CVPR 2025)
  • on improving visual prompting/object grounding without training (Wan et al., ECCV 2024)
  • on a more effective/efficient self-consistency method for LLM agents (Wang et al., ACL 2024)
  • on Western cultural bias in VLMs and the effect of pretraining language (Ananthram et al., ICLR 2025)
  • on visual commonsense in unimodal and multimodal models (Zhang et al, NAACL 2022)
  • on reasoning about occluded patterns (Pothiraj et al., 2025)

Before starting my Ph.D., I received my B.A.&Sc. with First Class Honours in Cognitive Science from McGill University, focusing in computer science and linguistics. While at McGill, I worked as a research assistant at the Montreal Language Modeling Lab (MLML), now MCQLL supervised by Morgan Sonderegger. I wrote my honours thesis (supervised by Timothy O’Donnell) on a variational inference algorithm for a model of language acquisition.