Responsible Use of AI in Systematic Reviews and Evidence Synthesis

Описание: Farhad Shokraneh prepared two similar sessions for the South West Public Health Development School, South West Population and Public Health Academy, NHS England. Presented on 22nd October 2025.

The session starts with information on AI terminology, types, classifications, myths, and confusions. It discusses rule-based automation, discriminative and predictive AI, and generative AI and large language models.

With a brief introduction to systematic reviews and types of evidence synthesis, it discusses practical tips and shows the examples of the responsible use of AI in each step of systematic reviews: formulating the research question, searching, screening, assessing the risk of bias, data extraction, meta-analysis, and assessing the certainty of evidence using GRADE. The webinar sums up with ideas on taking control of LLMs, their challenges, and risks, a final golden rule, and resources to look up.

00:00 Introducing the session and the speaker
00:50 Terminology: Intelligence, Learning, Cognition, Neural Networks, and Hallucinations
01:57 Three Myths and Confusions about AI: Topol Review, Types of AI (Discriminative vs Generative AI)
06:00 LLM Capabilities: What they can or can't do and relevance to systematic review
08:35 Myths and Confusions on Systematic Review
10:00 Systematic Reviewing Process and its Products: What is a Systematic Review and Types of Evidence Synthesis
16:16 Using LLMs in Protocol Development, Funding Unpublished Systematic Review, and Clinical Questions for Clinical Practice Guidelines
16:57 Using Types of AI in Search and Update Search Steps of Systematic Review
21:03 Using Discriminative AI in Record Screening or Title-Abstract Screening Step of Systematic Review
23:31 Why the Quality of Export File from Databases (=Input File to AI System) is Important: Example from Relevancy Ranking in Rayyan
25:36 Relevancy Ranking in Covidence
25:57 Priority Screening in ASReview Using Discriminative AI
27:01 PRIMSA 2020 Allows Us Exclude Irrelevant Records Using Classifiers (Discriminative AI): RCT Classifier, Automatic De-Duplication, and LLM in Covidence
28:18 Report Screening or Full Text Screening Using AI
28:53 Assessment of Risk of Bias using AI: Examples from Discriminative AI (RobotReviewer), Generative AI (LLM Chatbot Copilot GPT-4), and LLM-Embedded in Systematic Review Automation Programme (PICO Portal)
33:19 Data Extraction Using LLM
35:29 Tips for Data Extraction When Using LLM
38:29 Meta-Analysis Using LLM vs Rule-Based Automation
39:33 Example of Conducting Meta-Analysis and Visualizing a Forest Plot in Copilot (GPT-4)
40:19 Assessing the Certainty of Evidence (GRADE) by LLM
40:31 General Considerations When Using AI (Discriminative and Generative): Garbage In, Garbage Out, Human-In-The-Loop, Zero Trust, Documenting and Reporting, Context, Research vs Practice
43:00 What AI Tools or Types to Use and When: AI Embedded in Systematic Review Tools and AI as Assistant, Asking for Quotes/Evidence/Formula (Show Your Work), Repetition for Reproducibility
44:10 Taking Control of Large Language Models: Memory, Personality, Temperature Setting, Prompt Engineering, Orchestration, Create Your Agent, Connection to Apps, Learning and Archive Features
44:55 Why Using LLM is Challenging in Evidence Synthesis: Irreproducibility, Heterogeneity, Lack of Transparency, Contamination Bias
46:26 Risks of Using Large Language Models: Reduced Human Oversight, Forgotten Skills, Intellectual Property, No Guidance, Data Privacy, Environmental Impact, Algorithmic Bias, and Cybersecurity
48:28 Golden Rule in Using AI and Responsible Use of AI
48:45 References, Homework and Resources to Check
49:04 Future Developments: AI Literacy Course, Weekly Newsletter, YouTube Channel, and Living Map for Automation of Systematic Reviews (LEMASyR)
49:49 Questions and Answers: How do you appraise the use of AI versus human in peer-review, review and evidence synthesis: Options to Use AI Safely
52:19 Questions and Answers: Do Living Systematic Reviews get cited in research and does the conclusion of the review changes over time? Updating Search vs Updating Systematic Review
53:30 Questions and Answers: Is there any evidence of benefit for using AI in evidence synthesis and systematic reviews?
55:54 Questions and Answers: Will the materials from this webinar be available to the participants?