Past talk · AI/ML Seminar Series
Evaluating Foundational Models Using Insights from Their Pretraining Data
PhD Student, Department of Computer Science, University of California, Irvine
- Date & time
- Monday, November 25, 2024 · 1:00 PM
- Location
- Donald Bren Hall 4011
Abstract
Foundational models have demonstrated exceptional performance on established academic benchmarks, often narrowing the gap between human reasoning and artificial intelligence. While the success of these models is widely attributed to their scale, the critical role of pretraining data in shaping their capabilities and limitations is often acknowledged but rarely studied. In this talk, I will argue that understanding the true performance of foundational models requires going beyond conventional benchmark testing. In particular, incorporating insights from their pretraining data is essential for comprehensively evaluating and interpreting the models' capabilities and limitations. I show that while models often excel in benchmark settings, they can fail on basic, trivial reasoning tasks, raising concerns about their true robustness. This work cautions against overly optimistic interpretations of models' abilities based on canonical evaluation results.
About the speaker
Yasaman Razeghi is a final-year Ph.D. student at UCI, advised by Prof. Sameer Singh. She completed her master's and undergraduate studies at the University of Tehran in Iran. Her research focuses on understanding the relationships between pretraining data characteristics and model behavior. Most recently, she has been investigating foundational models in scenarios involving reasoning and multimodal capabilities.