Past talk · AI/ML Seminar Series
Training Precise Language Models for Imprecise Humans
Postdoctoral Researcher, Allen Institute for AI and University of Washington
- Date & time
- Monday, February 24, 2025 · 11:00 AM
- Location
- Donald Bren Hall 4011
Abstract
This talk examines methods for enhancing language model capabilities through post-training. While large language models have led to major breakthroughs in natural language processing, significant challenges persist due to the inherent ambiguity and underspecification in language. I will present a spectrum ranging from underspecification (preference modeling) to full specification (precise instruction following with verifiable constraints), and propose modeling approaches to increase language models' contextual robustness and precision. I will demonstrate how models can become more precise instruction followers through synthetic data, preference tuning, and reinforcement learning from verifiable rewards. On the preference data side, I will illustrate patterns of divergence in annotations, showing how disagreements stem from underspecification, and propose alternatives to the Bradley-Terry reward model for capturing pluralistic preferences. The talk concludes by connecting underspecification and reinforcement learning through a novel method: reinforced clarification question generation, which helps models obtain missing contextual information that is consequential for making predictions.
About the speaker
Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Yejin Choi. She is additionally supported by an Eric and Wendy Schmidt Postdoctoral Award. She obtained her PhD in Computer Science from the NLP lab at Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award. She holds an MSc from the University of Edinburgh and a BA from the University of Zurich.