Full Paper - The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Описание: This is a full reading of the paper: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

This paper finds that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that—when trained in isolation— reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

-----------------
You can find the full paper at: https://arxiv.org/abs/1803.03635
Note the paper has an appendix section which isn't covered in this video due to its length.

The paper’s author are Jonathan Frankle and Michael Carbin.
-----------------
About the Full Papers youtube channel:

Reading research papers takes a lot of time. This channel makes it easier.

You might enjoy this channel if you:
Prefer to learn through hearing rather than reading alone.
Like to take a first-pass through a paper while commuting.
Spend long days staring at a screen and want to give your eyes a rest, lay back and hear a research paper at the end of the day.
Have a backlog of research papers to read and would like to catch up by reading some of them more casually.
Want to take a first-pass through a paper before deciding if it’s worth sitting down and working through the details.

I tend to post papers I’m interested in, but if you have a paper you’d like me to generate a video for just send me a message on twitter at   / marcsto   or email me at [email protected] and I should be able to help.

This video was created by Marc Stogaitis. Just to make sure it’s clear, I did not write the paper, I’m just trying to help you learn it.

You can visit my website at https://marc.ai