AI-Assisted Peer Review at Scale The AAAI-26 AI Review Pilot with Joydeep Biswas

Описание: AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot

Presenter: Joydeep Biswas, Associate Professor, Department of Computer Science, The University of Texas at Austin, and Associate Director of Texas Robotics.

Abstract: Frontier multimodal language models are rapidly reshaping how we conduct and evaluate science. This talk presents the AAAI-26 AI review pilot, which explored a specific role for AI in the scientific process: peer review. In response to the explosive growth of AI publishing (over 30,000 initial submissions for AAAI-26) and the increasing technical capabilities of state-of-the-art language models, AAAI-26 ran a pilot in which every paper received one clearly labeled AI-generated review. No human reviewers were replaced, and final decisions remained entirely under human control.

I will describe what we built: a thorough multi-stage AI reviewing system that integrates multiple tools and techniques, with explicit criteria at each step, along with the infrastructure required to generate AI reviews for the full submission set in under 24 hours.

We also conducted an extensive voluntary survey of authors, reviewers, senior program committee members, and area chairs to assess and compare them with human reviews. Overall, respondents found the AI reviews helpful, and on average, they were preferred to human reviews across 6 of 9 criteria, including overall impressions, review focus, technical accuracy, and research suggestions. We also learned about the current limitations of AI in peer review.

I will close with lessons learned, opportunities for effective human-AI teaming in peer review, and open challenges in building and evaluating AI assistance for scientific reviewing.

Bio: Joydeep Biswas is an associate professor in the Department of Computer Science at the University of Texas at Austin and Associate Director of Texas Robotics. He leads the Autonomous Mobile Robotics Laboratory (AMRL), where he directs research focused on perception and planning for long-term autonomy in open-world settings. He is a recipient of the NSF CAREER award, the Amazon Research Award, and the JP Morgan Faculty Research Award, and serves as a Trustee of the RoboCup Federation and a Councilor of AAAI. He was an Associate Program Chair for AAAI-26 and led its AI-assisted peer review pilot.

Summary
Astronomical Data Processing Challenges
Joydeep talked about creating an article on a published work and the challenges of processing large astronomical datasets. Joydeep explained the role of CosmicAI members NRA and War Lab in handling data from sources like ALMA and DESI, and suggested contacting Stephanie Juneau for more information on using LLMs in this context. The conversation touched on the difficulties of incremental data processing and the need for efficient methods to handle large datasets in astronomy.
AI Peer Review Pilot Update
Joydeep introduced the third talk in CosmicAI's Spring 2026 Hybrid Seminar Series and discussed the AI-assisted peer review pilot at AAAI 26. He acknowledged the efforts of the team, including Chad, Matt, Kiri, and Jessy, in making this project possible. Joydeep also shared his interest in astrophotography and briefly mentioned the invited speaker Daniel Byteson for the Cosmic AI talk.
AI Peer Review Study Findings
Joydeep presented a study on using AI for scientific peer review, highlighting two main discoveries: a new planetary nebula candidate and challenges in AI-assisted peer review. He discussed the technical details of their AI review system, including preprocessing steps, model capabilities, and quality control measures. The study found that AI reviews were preferred to human reviews on 6 out of 9 criteria, though there were concerns about the environmental impact of running such models. Joydeep concluded that while AI reviews showed promise, they should be used as a supplement to human reviews rather than a replacement.
AI Review Capabilities and Development
Joydeep presented findings on AI review capabilities, noting that AI reviews were preferred by authors for 6 out of 9 criteria with lower variance compared to human reviews, though they were limited in understanding novelty and ranking severity. He emphasized that there is no single optimal review format, as different stakeholders have different needs.