Sara Hooker - The Hardware Lottery, Sparsity and Fairness

Описание: Dr. Tim Scarfe, Yannic Kilcher and Sayak Paul chat with Sara Hooker from the Google Brain team! We discuss her recent hardware lottery paper, pruning / sparsity, bias mitigation and intepretability.

The hardware lottery -- what causes inertia or friction in the marketplace of ideas? Is there a meritocracy of ideas or do the previous decisions we have made enslave us? Sara Hooker calls this a lottery because she feels that machine learning progress is entirely beholden to the hardware and software landscape. Ideas succeed if they are compatible with the hardware and software at the time and also the existing inventions. The machine learning community is exceptional because the pace of innovation is fast and we operate largely in the open, this is largely because we don't build anything physical which is expensive, slow and the cost of being scooped is high. We get stuck in basins of attraction based on our technology decisions and it's expensive to jump outside of these basins. So is this story unique to hardware and AI algorithms or is it really just the story of all innovation? Every great innovation must wait for the right stepping stone to be in place before it can really happen. We are excited to bring you Sara Hooker to give her take.

00:00:00 Sara joins
00:00:46 Show introduction with everyone on the call
00:03:40 Elevator pitch on hardware lottery
00:05:03 What's so special about hardware and tooling
00:06:51 Connectionist approaches losing out and now being stuck with them
00:09:53 GPU to TPU
00:15:03 Isn't this just a story of stepping stones and innovation in general (Kenneth Stanley reference)
00:18:22 We have a missing counterfactual of what hardware could exist
00:19:32 Capsule networks - we have converged on one "global update" paradigm of NNs
00:21:44 Compression / Structured / Unstructured sparsity
00:24:41 As you prune, what does a model forget? Longtail of model parameters encode low frequency information
00:28:09 Cultural Divide In Machine Learning (Welling vs Sutton)
00:31:23 Our own intelligence is based on local updates
00:33:28 Max Welling, Priors and the focussing on the long tail, lets not treat the data equally!
00:36:02 Sparsity training the future? (model compression research/gradient flow)
00:40:37 Is it a resource allocation problem? Too much exploration would spread us too thin
00:45:45 Isn't it just being at the right place at the right time? Is it a lottery?
00:49:03 National strategy to combat the hardware lottery
00:51:55 Ironic if DL created the next generation of hardware
00:52:23 Maybe we are in the AI winter now though? Maybe we need to go symbolic :)
00:56:08 Periods of feast and famine in ML research (Bayesian, causal, symbolic, DL etc)
00:57:44 Characterising and mitigating bias in compact models
01:01:13 Bias - dataset vs algorithm - how do we get rid of it, protected attributes
01:05:49 Are sparse networks more interpretable, single image interpretations, saliency methods
01:10:51 Protected attributes
01:14:14 How do you differentiate between something that is underrepresented vs more challenging
01:17:11 Anna Karenina principle and Sara's eccentric style of paper writing



Show notes; https://drive.google.com/file/d/1S_rH...
Sara Hooker page; https://www.sarahooker.me

Show notes -- https://drive.google.com/file/d/1S_rH...
Podcast version -- https://anchor.fm/machinelearningstre...
Yannic video on hardware lottery --    • The Hardware Lottery (Paper Explained)