Deep Learning Model Complexity: Concepts and Approaches (Part 1: Introduction)

Описание: Deep learning is disruptive in many applications mainly due to its superior performance. At the same time, many fundamental questions about deep learning remain unanswered. Model complexity of deep neural networks is one of them. Model complexity is concerned about how complicated a problem that a deep model can express and how nonlinear and complex the function of a model with given parameters can be.

In machine learning, data mining and deep learning, model complexity is always an important fundamental problem. Model complexity affects learnability of models on specific problems and data, as well as generalization ability of the model on unseen data. Moreover, the complexity of a learned model is affected not only by the model architecture itself, but also by the data distribution, data complexity, and information volume. In recent years, model complexity has become a more and more active direction, and has developed theoretical guiding significance in many areas, such as model architecture searching, graph representation, generalization study and model compression.

We propose this tutorial to overview the state-of-the-art research on deep learning model complexity. We summarize the model complexity studies into two directions: model expressive capacity and effective model complexity, and review the latest progress on these two directions.In addition, we introduce some application examples of deep learning model complexity to demonstrate its utility.