BEGIN:VCALENDAR VERSION:2.0 PRODID:-//UC Irvine//CML Seminars//EN CALSCALE:GREGORIAN METHOD:PUBLISH X-WR-CALNAME:Training Precise Language Models for Imprecise Humans X-WR-TIMEZONE:America/Los_Angeles BEGIN:VTIMEZONE TZID:America/Los_Angeles BEGIN:DAYLIGHT TZOFFSETFROM:-0800 TZOFFSETTO:-0700 TZNAME:PDT DTSTART:20070311T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:-0700 TZOFFSETTO:-0800 TZNAME:PST DTSTART:20071104T020000 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:2025-02-24-valentina-pyatkin@cml.ics.uci.edu DTSTAMP:20250224T000000Z SEQUENCE:57690 DTSTART;TZID=America/Los_Angeles:20250224T110000 DTEND;TZID=America/Los_Angeles:20250224T120000 SUMMARY:[CML Seminar] Valentina Pyatkin: Training Precise Language Models f or Imprecise Humans LOCATION:Donald Bren Hall 4011 DESCRIPTION:Valentina Pyatkin\, Postdoctoral Researcher\, Allen Institute f or AI and University of Washington\n\nTitle: Training Precise Language Mod els for Imprecise Humans\n\nAbstract: This talk examines methods for enhan cing language model capabilities through post-training. While large langua ge models have led to major breakthroughs in natural language processing\, significant challenges persist due to the inherent ambiguity and underspe cification in language. I will present a spectrum ranging from underspecif ication (preference modeling) to full specification (precise instruction f ollowing with verifiable constraints)\, and propose modeling approaches to increase language models' contextual robustness and precision. I will dem onstrate how models can become more precise instruction followers through synthetic data\, preference tuning\, and reinforcement learning from verif iable rewards. On the preference data side\, I will illustrate patterns of divergence in annotations\, showing how disagreements stem from underspec ification\, and propose alternatives to the Bradley-Terry reward model for capturing pluralistic preferences. The talk concludes by connecting under specification and reinforcement learning through a novel method: reinforce d clarification question generation\, which helps models obtain missing co ntextual information that is consequential for making predictions.\n\nhttp s://cml.ics.uci.edu/seminars/2025-02-24-valentina-pyatkin X-ALT-DESC;FMTTYPE=text/html:Valentina Pyatkin\, Postdoc toral Researcher\, Allen Institute for AI and University of Washington

Title: Training Precise Language Models for Imprecise Humans

Abstract: This talk examines methods for enhancing language mo del capabilities through post-training. While large language models have l ed to major breakthroughs in natural language processing\, significant cha llenges persist due to the inherent ambiguity and underspecification in la nguage. I will present a spectrum ranging from underspecification (prefere nce modeling) to full specification (precise instruction following with ve rifiable constraints)\, and propose modeling approaches to increase langua ge models' contextual robustness and precision. I will demonstrate how mod els can become more precise instruction followers through synthetic data\, preference tuning\, and reinforcement learning from verifiable rewards. O n the preference data side\, I will illustrate patterns of divergence in a nnotations\, showing how disagreements stem from underspecification\, and propose alternatives to the Bradley-Terry reward model for capturing plura listic preferences. The talk concludes by connecting underspecification an d reinforcement learning through a novel method: reinforced clarification question generation\, which helps models obtain missing contextual informa tion that is consequential for making predictions.

https://cml.ics.uc i.edu/seminars/2025-02-24-valentina-pyatkin URL:https://cml.ics.uci.edu/seminars/2025-02-24-valentina-pyatkin END:VEVENT END:VCALENDAR