Automatic Differentiation and SciML: What Can Go Wrong | Chris Rackauckas | JuliaHEP 2023

Описание: Title:
Automatic Differentiation and SciML: What Can Go Wrong, and What to Do About It?

Scientific machine learning (SciML) through differentiable simulation is a method that is faster and more robust than techniques like physics-informed neural networks (PINNs), neural operators, and other techniques. However, doing differentiable simulation correctly requires a deep understanding of automatic differentiation and the numerical properties of simulation. In this talk, we dive deep into the numerical stability of derivatives of simulation processes, showing how naive applications of differentiable programming can give surprisingly incorrect results, and how one may need to modify simulations in order to perform robust automated model discovery and calibration.

Originally part of the JuliaHEP 2023 Workshop at the ECAP (Erlangen Centre for Astroparticle Physics), https://indico.cern.ch/event/1292759/

Bio: Dr. Chris Rackauckas is the VP of Modeling and Simulation at JuliaHub, the Director of Scientific Research at Pumas-AI, Co-PI of the Julia Lab at MIT, and the lead developer of the SciML Open Source Software Organization. For his work in mechanistic machine learning, his work is credited for the 15,000x acceleration of NASA Launch Services simulations and recently demonstrated a 60x-570x acceleration over Modelica tools in HVAC simulation, earning Chris the US Air Force Artificial Intelligence Accelerator Scientific Excellence Award. See more at https://chrisrackauckas.com/. He is the lead developer of the Pumas project and has received a top presentation award at every ACoP in the last 3 years for improving methods for uncertainty quantification, automated GPU acceleration of nonlinear mixed effects modeling (NLME), and machine learning assisted construction of NLME models with DeepNLME. For these achievements, Chris received the Emerging Scientist award from ISoP.

Resources
Example presented by Chris Rackauckas on Discourse: https://discourse.julialang.org/t/doe...

Contents
0:00:00 Welcome
0:00:45 Content outline
0:01:50 Prologue: Why do differentiable simulation?
0:02:32 Universal Approximation Theorem
0:06:00 UODE example 1: infection model
0:15:50 Why neural networks vs other universal approximators
0:20:36 UODE example 2: learning binary black hole dynamics from LIGO data
0:22:50 UODE example 3: diffusion-advection process in a chemical reactor system
0:28:40 Scientific machine learning digital twins
0:32:46 Does scientific machine learning require differentiation of the simulator?
0:33:03 UODE example 4: ocean columns for climate models
0:39:16 Integral control to prevent solution drift
0:41:20 Differentiation of solvers and automatic differentiation
0:55:40 Three steps to summarize the solution process
1:02:16 Why adjoints by reversing is unconditionally unstable
1:08:30 What is automatic differentiation and how does it help?
1:09:02 Worked example of automatic differentiation (see in Resource cathegory for a link)
1:12:32 Dual numbers and automatic differentiation
1:20:58 What does automatic differentiation of an ODE solver give you?
1:31:40 When automatic differentiation gives numerically incorrect answers
1:35:42 Benefits of adaptivity
1:48:14 Other cases where automatic differentiation can fail (e.g., chaotic systems)
1:57:33 SciML common interface for Julia equation solvers
2:02:50 Returning to binary black hole dynamics as a worked example of successful SciML
2:08:11 Methods to improve the fitting process and pitfalls of single shooting
2:15:45 Multiple shooting and collocation
2:24:17 Neural network architectures in ODEs
2:35:11 Other methods that ignore derivative issues and future directions
2:45:23 Reservoir computing
2:47:52 Final comments and questions

S/O to https://github.com/agchesebro for the video timestamps!

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/JuliaCommunity/You...

Interested in improving the auto generated captions? Get involved here: https://github.com/JuliaCommunity/You...