Simple reverse-mode Autodiff in Python

Описание: Ever wanted to know how automatic differentiation (the general case of backpropagation for training neural networks in deep learning) works? Let's have an easy tutorial in Python. Here is the code: https://github.com/Ceyron/machine-lea...

-----

👉 This educational series is supported by the world-leaders in integrating machine learning and artificial intelligence with simulation and scientific computing, Pasteur Labs and Institute for Simulation Intelligence. Check out https://simulation.science/ for more on their pursuit of 'Nobel-Turing' technologies (https://arxiv.org/abs/2112.03235 ), and for partnership or career opportunities.

-------

📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): https://github.com/Ceyron/machine-lea...

📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff:   / felix-koehler   and   / felix_m_koehler  

💸 : If you want to support my work on the channel, you can become a Patreon here:   / mlsim  

🪙: Or you can make a one-time donation via PayPal: https://www.paypal.com/paypalme/Felix...

-------

Timestamps:
00:00 Intro
00:18 Our simple (unary) function
00:35 Closed-Form symbolic derivative
01:16 Validate derivative by finite differences
01:55 What is automatic differentiation?
02:53 Backprop rule for sine function
04:27 Backprop rule for exponential function
06:06 Rule library as a dictionary
07:14 The heart: forward and backward pass
11:17 Trying the rough autodiff interface
13:15 Syntactic sugar to get a high-level interface
14:10 Compare autodiff with symbolic differentiation
14:59 Outro