publications

2024

1. arxiv

   Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

   David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal

   arXiv 2024

   PDF
2. arxiv

   Language-guided Skill Learning with Temporal Variational Inference

   Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin, George Konidaris, and 3 more authors

   arXiv 2024

   PDF
3. arxiv

   Soft Self-Consistency Improves Language Model Agents

   Han Wang*, Archiki Prasad*, Elias Stengel-Eskin*, and Mohit Bansal

   arXiv 2024

   PDF
4. arxiv

   GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

   Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, and 5 more authors

   arXiv 2024

   PDF
5. arxiv

   MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models

   Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal

   arXiv 2024

   PDF
6. arxiv

   ReGAL: Refactoring Programs to Discover Generalizable Abstractions

   Elias* Stengel-Eskin, Archiki* Prasad, and Mohit Bansal

   arXiv 2024

   PDF
7. ICLR

   Zero and Few-shot Semantic Parsing with Ambiguous Inputs

   Elias Stengel-Eskin, Kyle Rawlins, and Benjamin Van Durme

   ICLR 2024

   PDF
8. ICLR

   Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models

   Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal

   ICLR 2024

   PDF

2023

1. EMNLP

   Did You Mean...? Confidence-based Trade-offs in Semantic Parsing

   Elias Stengel-Eskin, and Benjamin Van Durme

   EMNLP 2023

   PDF
2. TACL

   Calibrated Interpretation: Confidence Estimation in Semantic Parsing

   Elias Stengel-Eskin, and Benjamin Van Durme

   TACL 2023

   PDF
3. ACL

   Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA

   Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme

   ACL 2023

   PDF
4. CVPR

   Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning (CVPR Highlight)

   Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, and 3 more authors

   CVPR 2023

   PDF

2022

1. MASCSLL

   Automatic Evaluation of Chit-chat via Semantic Parsing

   Shalaka Vaidya, Elias Stengel-Eskin, and João Sedoc

   Mid-Atlantic Student Colloquium on Speech, Language and Learning 2022

   Abs

   With the increasing popularity and capabilities of conversational dialog systems, there is a need for metrics that are reliable, robust and automatic that facilitate model comparison without expensive human intervention. We propose an automatic, semantically-grounded, and domain-independent metric requiring no humans in the loop. This metric uses Gricean Maxims to parameterize the quality of dialog across various utterances. Finally, we evaluate the calculated metric with the breakdown markers in human-chatbot conversations.

2. EMNLP

   The Curious Case of Control

   Elias Stengel-Eskin, and Benjamin Van Durme

   Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing 2022

   Abs PDF

   Children acquiring English make systematic errors on subject control sentences even after they have reached near-adult competence (C. Chomsky, 1969), possibly due to heuristics based on semantic roles (Maratsos, 1974). Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other. We find that models can be categorized by behavior into three separate groups, with broad differences between the groups. The outputs of models in the largest group are consistent with positional heuristics that succeed on subject control but fail on object control. This result is surprising, given that object control is orders of magnitude more frequent in the text data used to train such models. We examine to what degree the models are sensitive to prompting with agent-patient information, finding that raising the salience of agent and patient relations results in significant changes in the outputs of most models. Based on this observation, we leverage an existing dataset of semantic proto-role annotations (White, et al. 2020) to explore the connections between control and labeling event participants with properties typically associated with agents and patients.

3. EMNLP

   When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systems

   Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, and 4 more authors

   Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing 2022

   Abs PDF

   In natural language understanding (NLU) production systems, users’ evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space. This requires additional training data and results in ever-growing datasets. We present the first systematic investigation into this incremental symbol learning scenario. Our analyses reveal a troubling quirk in building (broad-coverage) NLU systems: as the training dataset grows, more data is needed to learn new symbols, forming a vicious cycle. We show that this trend holds for multiple mainstream models on two common NLU tasks: intent recognition and semantic parsing. Rejecting class imbalance as the sole culprit, we reveal that the trend is closely associated with an effect we call source signal dilution, where strong lexical cues for the new symbol become diluted as the training dataset grows. Selectively dropping training examples to prevent dilution often reverses the trend, showing the over-reliance of mainstream neural NLU models on simple lexical cues and their lack of contextual understanding.

4. NAACL

   Visual Commonsense in Pretrained Unimodal and Multimodal Models

   Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel-Eskin

   In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Jul 2022

   Abs PDF

   Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world-knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.

2021

1. CoRL

   Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions

   Elias Stengel-Eskin*, Andrew Hundt*, Zhuohong He, Aditya Murali, and 3 more authors

   In 5th Annual Conference on Robot Learning Jul 2021

   Abs PDF

   Enabling human operators to interact with robotic agents using natural language would allow non-experts to intuitively instruct these agents. Towards this goal, we propose a novel Transformer-based model which enables a user to guide a robot arm through a 3D multi-step manipulation task with natural language commands. Our system maps images and commands to masks over grasp or place locations, grounding the language directly in perceptual space. In a suite of block rearrangement tasks, we show that these masks can be combined with an existing manipulation framework without re-training, greatly improving learning efficiency. Our masking model is several orders of magnitude more sample efficient than typical Transformer models, operating with hundreds, not millions, of examples. Our modular design allows us to leverage supervised and reinforcement learning, providing an easy interface for experimentation with different architectures. Our model completes block manipulation tasks with synthetic commands 530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning. We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourced language to a simulated environment. Our method’s 25.3% absolute improvement in identifying the correct block on the ported dataset demonstrates its ability to handle syntactic and lexical variation.

2. ICCV

   Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images

   Zhuowan Li, Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, and 3 more authors

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Oct 2021

   Abs PDF

   While neural symbolic methods demonstrate impressive performance in visual question answering on synthetic images, their performance suffers on real images. In this paper, we identify that the long-tail distribution of visual concepts and unequal importance of reasoning steps in real data are the two key obstacles that limit the models’ real-world potentials. To address these challenges, we propose a new paradigm, Calibrating Concepts and Operations (CCO), which enables neural symbolic models to capture underlying data characteristics and to reason with hierarchical importance. Specifically, we introduce an executor with learnable concept embedding magnitudes for handling distribution imbalance, and an operation calibrator for highlighting important operations and suppressing redundant ones. Our experiments show CCO substantially boosts the performance of neural symbolic methods on real images. By evaluating models on the real world dataset GQA, CCO helps the neural symbolic method NSCL outperforms its vanilla counterpart by a large margin of 9.1% (from 47.0% to 56.1%), which also greatly reduces the performance gap between symbolic and non-symbolic methods. Additionally, we create a perturbed test set for better understanding and analyzing model performance on real images.

3. UnImplicit

   Human-Model Divergence in the Handling of Vagueness

   Elias Stengel-Eskin, Jimena Guallar-Blasco, and Benjamin Van Durme

   In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language Aug 2021

   Abs PDF

   While aggregate performance metrics can generate valuable insights at a large scale, their dominance means more complex and nuanced language phenomena, such as vagueness, may be overlooked. Focusing on vague terms (e.g. sunny, cloudy, young, etc.) we inspect the behavior of visually grounded and text-only models, finding systematic divergences from human judgments even when a model’s overall performance is high. To help explain this disparity, we identify two assumptions made by the datasets and models examined and, guided by the philosophy of vagueness, isolate cases where they do not hold.

4. TACL

   Joint Universal Syntactic and Semantic Parsing

   Elias Stengel-Eskin, Kenton Murray, Sheng Zhang, Aaron Steven White, and 1 more author

   Transactions of the Association for Computational Linguistics Aug 2021

   Abs PDF

   While numerous attempts have been made to jointly parse syntax and semantics, high performance in one domain typically comes at the price of performance in the other. This trade-off contradicts the large body of research focusing on the rich interactions at the syntax-semantics interface. We explore multiple model architectures which allow us to exploit the rich syntactic and semantic annotations contained in the Universal Decompositional Semantics (UDS) dataset, jointly parsing Universal Dependencies and UDS to obtain state-of-the-art results in both formalisms. We analyze the behaviour of a joint model of syntax and semantics, finding patterns supported by linguistic theory at the syntax-semantics interface. We then investigate to what degree joint modeling generalizes to a multilingual setting, where we find similar trends across 8 languages.

5. SCiL

   Exploring Human-Model Divergence Through Vagueness

   Elias Stengel-Eskin, Jimena Guallar-Blasco, and Benjamin Van Durme

   Proceedings of the Society for Computation in Linguistics Feb 2021

   Abs PDF

   While aggregate performance metrics can generate valuable insights at a large scale, their dominance means more complex and nuanced language phenomena, such as vagueness, may be overlooked. Focusing on vague terms (e.g. sunny, cloudy, young, etc.) we inspect the behavior of visually grounded and text-only models, finding systematic divergences from human judgments even when a model’s overall performance is high. To help explain this disparity, we identify two assumptions made by the datasets and models examined and, guided by the philosophy of vagueness, isolate cases where they do not hold.

6. TACL

   Iterative Paraphrastic Augmentation with Discriminative Span Alignment

   Ryan Culkin, J. Edward Hu, Elias Stengel-Eskin, Guanghui Qin, and 1 more author

   Transactions of the Association for Computational Linguistics May 2021

   Abs PDF

   We introduce a novel paraphrastic augmentation strategy based on sentence-level lexically constrained paraphrasing and discriminative span alignment. Our approach allows for the large-scale expansion of existing datasets or the rapid creation of new datasets using a small, manually produced seed corpus. We demonstrate our approach with experiments on the Berkeley FrameNet Project, a large-scale language understanding effort spanning more than two decades of human labor. With four days of training data collection for a span alignment model and one day of parallel compute, we automatically generate and release to the community 495,300 unique (Frame,Trigger) pairs in diverse sentential contexts, a roughly 50-fold expansion atop FrameNet v1.7. The resulting dataset is intrinsically and extrinsically evaluated in detail, showing positive results on a downstream task.

2020

1. ACL

   Universal Decompositional Semantic Parsing

   Elias Stengel-Eskin, Aaron Steven White, Sheng Zhang, and Benjamin Van Durme

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics May 2020

   Abs PDF Slides

   We introduce a transductive model for parsing into Universal Decompositional Semantics (UDS) representations, which jointly learns to map natural language utterances into UDS graph structures and annotate the graph with decompositional semantic attribute scores. We also introduce a strong pipeline model for parsing into the UDS graph structure, and show that our transductive parser performs comparably while additionally performing attribute prediction. By analyzing the attribute prediction errors, we find the model captures natural relationships between attribute groups.

2. LREC

   The Universal Decompositional Semantics Dataset and Decomp Toolkit

   Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, and 7 more authors

   In Proceedings of The 12th Language Resources and Evaluation Conference May 2020

   Abs PDF

   We present the Universal Decompositional Semantics (UDS) dataset (v1. 0), which is bundled with the Decomp toolkit (v0. 1). UDS1. 0 unifies five high-quality, decompositional semantics-aligned annotation sets within a single semantic graph specification—with graph structures defined by the predicative patterns produced by the PredPatt tool and real-valued node and edge attributes constructed using sophisticated normalization procedures. The Decomp toolkit provides a suite of Python 3 tools for querying UDS graphs using SPARQL. Both UDS1. 0 and Decomp0. 1 are publicly available at this http URL.

2019

1. EMNLP

   A Discriminative Neural Model for Cross-Lingual Word Alignment

   Elias Stengel-Eskin, Tzu-Ray Su, Matt Post, and Benjamin Van Durme

   In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) May 2019

   Abs PDF

   We introduce a novel discriminative word alignment model, which we integrate into a Transformer-based machine translation model. In experiments based on a small number of labeled examples (  1.7 K-5K sentences) we evaluate its performance intrinsically on both English-Chinese and English-Arabic alignment, where we achieve major improvements over unsupervised baselines (11-27 F1). We evaluate the model extrinsically on data projection for Chinese NER, showing that our alignments lead to higher performance when used to project NER tags from English to Chinese. Finally, we perform an ablation analysis and an annotation experiment that jointly support the utility and feasibility of future manual alignment elicitation.

2017

1. Interspeech

   Polyglot and Speech Corpus Tools: A System for Representing, Integrating, and Querying Speech Corpora.

   Michael McAuliffe, Elias Stengel-Eskin, Michaela Socolof, and Morgan Sonderegger

   In INTERSPEECH May 2017

   Abs PDF

   Speech datasets from many languages, styles, and sources exist in the world, representing significant potential for scientific studies of speech—particularly given structural similarities among all speech datasets. However, studies using multiple speech corpora remain difficult in practice, due to corpus size, complexity, and differing formats. We introduce open-source software for unified corpus analysis: integrating speech corpora and querying across them. Corpora are stored in a custom ‘polyglot persistence’scheme that combines three sub-databases mirroring different data types: a Neo4j graph database to represent temporal annotation graph structure, and SQL and InfluxDB databases to represent meta-and acoustic data. This scheme abstracts away from the idiosyncratic formats of different speech corpora, while mirroring the structure of different data types improves speed and scalability. A Python API and a GUI both allow for: enriching the database with positional, hierarchical, temporal, and signal measures (eg utterance boundaries, f0) that are useful for linguistic analysis; querying the database using a simple query language; and exporting query results to standard formats for further analysis. We describe the software, summarize two case studies using it to examine effects on pitch and duration across languages, and outline planned future development.