DS-GA 3001 Modern Topics in Statistical Learning Theory

DS-GA 3001 Special Topics in Data Science:
Modern Topics in Statistical Learning Theory

Term: Spring 2023
Instructor: Dr. Qi Lei
Level: Graduate / PhD

Topics

Basics in Machine Learning;

Generalization bound: concentration inequality, uniform convergence, complexity measure;

Theory of deep learning: optimization, neural tangent kernel, implicit/algorithmic regularization;

Transfer learning: Meta learning, Domain adaptation, Domain generalization;

Self-supervised learning: Data augmentation, Self-supervised learning, Transformer;

Description

This course is a graduate-level topic course focusing on the theoretical grounding and statistical properties of the modern learning algorithms — with a focus on weakly supervised learning.

The intended topics to cover include: basics in machine learning, optimization and generalization bound, followed by the introduction and theoretical understanding surrounding meta-learning, self-supervised learning, and domain adaptation.

To benefit from this class, strong linear algebra, probability, and optimization background are required. Students should be familiar with basic machine learning and deep learning concepts.

The class consists of 3 units. In the first unit, we will cover the more standard theoretical analysis tools used in deep learning including stochastic gradient, uniform convergence theory and statistical learning theory.

After the first unit, the course will move on to specific topics (Unit 2: transfer learning and 3: selfsupervised learning). Since some topics covered in this course are quite recent, some content will be based on recent papers instead of a textbook.