Tackling challenges in scaling test-time compute

Описание: Suhas Pai, CTO, Hudson Labs



About the Speaker:
Suhas Pai is a NLP researcher and co-founder/CTO at Hudson Labs, a Toronto based Y-combinator backed startup. He is the author of the book 'Designing Large Language Model Applications', published by O'Reilly Media. He has contributed to the development of several open-source LLMs, including being the co-lead of the Privacy working group at BigScience, as part of the BLOOM LLM project. Suhas is active in the ML community, being Chair of the TMLS (Toronto Machine Learning Summit) conference since 2021. He is also a frequent speaker at AI conferences worldwide, and hosts regular seminars discussing the latest research in the field of NLP.


Abstract:
Reasoning models like OpenAI's o3 and DeepSeek's R1 herald a new paradigm that leverages test-time compute to solve tasks requiring reasoning. These models represent a departure from traditional LLMs, upending long-held assumptions about them. In this session, we will introduce the concept of test-time computation and discuss the different dimensions along which test-time computation can be expended and scaled. We will demonstrate the key limitations of current test-time scaling paradigms, touching upon topics like thought budgeting, context management, and confidence estimation, and share potential solutions for tackling them. Finally, we will showcase best practices for prompting reasoning models.