BEGIN:VCALENDAR VERSION:2.0 PRODID:-//UC Irvine//CML Seminars//EN CALSCALE:GREGORIAN METHOD:PUBLISH X-WR-CALNAME:Evaluating Foundational Models Using Insights from Their Pretr aining Data X-WR-TIMEZONE:America/Los_Angeles BEGIN:VTIMEZONE TZID:America/Los_Angeles BEGIN:DAYLIGHT TZOFFSETFROM:-0800 TZOFFSETTO:-0700 TZNAME:PDT DTSTART:20070311T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:-0700 TZOFFSETTO:-0800 TZNAME:PST DTSTART:20071104T020000 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:2024-11-25-yasaman-razeghi@cml.ics.uci.edu DTSTAMP:20241125T000000Z SEQUENCE:57690 DTSTART;TZID=America/Los_Angeles:20241125T130000 DTEND;TZID=America/Los_Angeles:20241125T140000 SUMMARY:[CML Seminar] Yasaman Razeghi: Evaluating Foundational Models Using Insights from Their Pretraining Data LOCATION:Donald Bren Hall 4011 DESCRIPTION:Yasaman Razeghi\, PhD Student\, Department of Computer Science\ , University of California\, Irvine\n\nTitle: Evaluating Foundational Mode ls Using Insights from Their Pretraining Data\n\nAbstract: Foundational mo dels have demonstrated exceptional performance on established academic ben chmarks\, often narrowing the gap between human reasoning and artificial i ntelligence. While the success of these models is widely attributed to the ir scale\, the critical role of pretraining data in shaping their capabili ties and limitations is often acknowledged but rarely studied. In this tal k\, I will argue that understanding the true performance of foundational m odels requires going beyond conventional benchmark testing. In particular\ , incorporating insights from their pretraining data is essential for comp rehensively evaluating and interpreting the models' capabilities and limit ations. I show that while models often excel in benchmark settings\, they can fail on basic\, trivial reasoning tasks\, raising concerns about their true robustness. This work cautions against overly optimistic interpretat ions of models' abilities based on canonical evaluation results.\n\nhttps: //cml.ics.uci.edu/seminars/2024-11-25-yasaman-razeghi X-ALT-DESC;FMTTYPE=text/html:Yasaman Razeghi\, PhD Stude nt\, Department of Computer Science\, University of California\, Irvine

Title: Evaluating Foundational Models Using Insights from Thei r Pretraining Data

Abstract: Foundational models have demons trated exceptional performance on established academic benchmarks\, often narrowing the gap between human reasoning and artificial intelligence. Whi le the success of these models is widely attributed to their scale\, the c ritical role of pretraining data in shaping their capabilities and limitat ions is often acknowledged but rarely studied. In this talk\, I will argue that understanding the true performance of foundational models requires g oing beyond conventional benchmark testing. In particular\, incorporating insights from their pretraining data is essential for comprehensively eval uating and interpreting the models' capabilities and limitations. I show t hat while models often excel in benchmark settings\, they can fail on basi c\, trivial reasoning tasks\, raising concerns about their true robustness . This work cautions against overly optimistic interpretations of models' abilities based on canonical evaluation results.

https://cml.ics.uci.ed u/seminars/2024-11-25-yasaman-razeghi URL:https://cml.ics.uci.edu/seminars/2024-11-25-yasaman-razeghi END:VEVENT END:VCALENDAR