MATH-GA 2840 Theory of Deep Learning

MATH-GA 2840 Advanced Topics in Applied Math:
Theory of Deep Learning

Term: Spring 2023
Instructor: Dr. Arthur Jacot-Guillarmod
Level: PhD

Topics

Neural Tangent Kernel; Spectral Bias of DNNs; Generalization; Curse of dimensionality; Leaving the NTK regime; Active regimes of diagonal linear networks; Fully-connected linear networks; Shallow networks; convex reformulation; Mean-field limit; Deep Networks;

Description

An overview of the many distinct training dynamics of Deep Neural Networks (DNNs) and their impact on the function learned by the network. The focus will mostly be on mathematical analysis of deep networks with a large number of neurons. Topics covered: the Neural Tangent Kernel and related dynamics; the spectral bias of DNNs; the transition between lazy and active regimes; the implicit bias of DNNs under the cross-entropy loss or with L2-regularization; Saddle-to-Saddle dynamics; appearance of sparsity for different DNN architectures.

Prerequisites: Linear Algebra, Multi-dimensional analysis (e.g. gradients, Hessians), Probability Theory. Recommended but not required: experience training DNNs, high dimensional probability.