Prof. Eric Xin Wang: Building AI Agents that Reason and Act Like Humans

Описание: Talk Abstract: In this talk, I will present our recent progress toward building general-purpose AI agents that can reason and act like humans, from automating complex computer tasks to engaging in intuitive, grounded reasoning. We begin with Agent S, an open agentic framework for autonomous interaction with Graphical User Interfaces (GUIs). Agent S addresses three key challenges in real-world task automation: acquiring domain-specific knowledge, planning over long horizons, and adapting to diverse, dynamic interfaces. It introduces experience-augmented hierarchical planning, combining external web search with internal memory retrieval to support efficient task decomposition and execution. Our latest version, Agent S3, achieves new state-of-the-art performance with Behavior Best-of-N to scale agents for computer use, approaching human-level AI that can serve as capable virtual workers in everyday and enterprise settings. To complement these agentic capabilities, we explore advances in reasoning. We propose the GRIT method to teach Multimodal Large Language Models (MLLMs) to "think with images", interleaving text tokens and visual anchors throughout the reasoning process. We also introduce Soft Thinking, a training-free paradigm that supports fluid, abstract thought, enabling models to explore multiple possibilities in parallel rather than committing to one rigid step at a time. Together, these lines of work lay the foundation for AI agents that can see, think, and act reliably in complex, real-world environments.

Bio: Xin (Eric) Wang is an Assistant Professor in The Computer Science Department at UC Santa Barbara and the Head of Research at Simular. His research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on Multimodal and Embodied AI Agents. Previously, he was a faculty at UC Santa Cruz and also spent time at Google Research, Meta FAIR, Microsoft Research, and Adobe Research. Eric has served as (Senior) Area Chair for conferences such as ACL, NAACL, EMNLP, ICLR, and NeurIPS, and organized workshops and tutorials at those venues. He has received several awards and recognitions for his work, including Best Paper Awards from CVPR and ICLRW, Google Research Faculty Award, Amazon Alexa Prize Awards, Cisco Research Award, eBay Research Awards, and various gift awards from Adobe, Snap, Microsoft, Cybever, etc.