Séminaire ayant lieu dans le cadre du Doctorat en informatique cognitive, en collaboration avec le centre de recherche CRIA       

TITRE : Embodied Language: Evaluating LLMs in the Real World

Yonatan BISK

Jeudi le 30 octobre 2025 à 10h30

Local PK-5115 (Il est possible d'y assister en virtuel en vous inscrivant ici)       

RÉSUMÉ

This talk examines the critical challenge of evaluating Large Language Models in interactive, embodied settings where language must connect to physical actions and environmental understanding. Drawing from recent research in embodied AI and language grounding, I will explore how current LLMs perform when tasked with interpreting language instructions that require spatial reasoning, object manipulation, and social interaction. The discussion will cover methodological frameworks for assessing language-to-action capabilities, including benchmarks that move beyond traditional text-based evaluation to encompass multimodal environments where language commands must be translated into executable actions. The talk will address fundamental questions about what it means for AI systems to truly understand language in the context of physical agency, examining both the successes and systematic failures of LLMs in interactive settings that require grounded communication and sensorimotor integration.

BIOGRAPHIE

Yonatan BISK is Assistant Professor at Carnegie Mellon University's Language Technologies Institute and Robotics Institute, where he founded the REAL Center (Robotics, Embodied AI, and Learning). His research focuses on grounded and embodied natural language processing, exploring how language interacts with vision, action, and reasoning in physical environments. Bisk earned his PhD from the University of Illinois at Urbana-Champaign in unsupervised grammar induction and held postdoctoral positions at USC's Information Sciences Institute, University of Washington, and Allen Institute for AI. He has been a visiting researcher at Microsoft Research and Meta AI. He teaches courses on "Talking to Robots" and "Multimodal Machine Learning."

RÉFÉRENCES

Mecattaf, M. G., Slater, B., Tešić, M., Prunty, J., Voudouris, K., & Cheke, L. G. (2024). A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment. arXiv.

Wu, Y., Min, S. Y., Bisk, Y., Salakhutdinov, R., Azaria, A., Li, Y., Mitchell, T., & Prabhumoye, S. (2023). Plan, Eliminate, and Track – Language Models are Good Teachers for Embodied Agents. arXiv.

Bisk, Y., Zellers, R., Gao, J., & Choi, Y. (2020). PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence, 34, 7432–7439.

Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020). ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Bisk, Y., Holtzman, A., Thomason, J., et al. (2020). Experience Grounds Language. EMNLP.