Quicklinks: Back to my homepage

Description

This textbook on the mathematics of data has two intended audiences:

• For students majoring in math (or other quantitative fields like physics, economics, engineering, etc.): it is meant as an invitation to data science and AI from a rigorous mathematical perspective.
• For (mathematically-inclined) students in data science related fields (at the undergraduate or graduate level): it can serve as a mathematical companion to machine learning, AI, and statistics courses.

It is based on Jupyter notebooks that were developed for MATH 535: MATHEMATICAL METHODS IN DATA SCIENCE, a one-semester advanced undergraduate and Master's level course offered at UW-Madison.

A print version of the book will be published by Cambridge University Press.

Course Information (Spring 2025)

• Course: MATH 535: Mathematical Methods in Data Science
• Instructor: Sebastien Roch
• Lectures: MoWeFr 11:00AM - 11:50AM
• Location: SOC SCI 6102
• Office Hours: TBA
• Prerequisites: (MATH 320, 340, 341, 375 or COMP SCI/E C E/M E 532) and (MATH/STAT 309, 431, MATH 531, STAT 311 or E C E 331) and (MATH 322, 341, 375, 421, 467, or COMP SCI 577), graduate/professional standing, or member of Pre-Masters Mathematics (Visiting Intl) Prgrm

Course Schedule:

Online book and Jupyter notebooks

Textbook: Current version of the full MMiDS book

Links to specific chapters are below, together some additional materials (assignments, Jupyter notebooks, datasets, auto-quizzes, etc.). Most of these resources are also available on the GitHub page of the book.

Python package: To run some of the code below, you will need mmids.py.

Chap 1: Introduction

• 1.1. Motivating example: identifying penguin species
• 1.2. Background: quick refresher of matrix algebra, differential calculus, and elementary probability
• 1.3. Clustering: an objective, an algorithm and a guarantee
• 1.4. Some observations about high-dimensional data
• 1.5. Exercises
• 1.6. Online supplementary material
• Data: penguins-measurements.csv, penguins-species.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 2: Least squares: geometric, algebraic, and numerical aspects

• 2.1. Motivating example: predicting sales
• 2.2. Background: review of vector spaces and matrix inverses
• 2.3. Geometry of least squares: the orthogonal projection
• 2.4. QR decomposition and Householder transformations
• 2.5. Application to regression analysis
• 2.6. Exercises
• 2.7. Online supplementary material
• Data: advertising.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 3: Optimization theory and algorithms

• 3.1. Motivating example: analyzing customer satisfaction
• 3.2. Background: review of differentiable functions of several variables
• 3.3. Optimality conditions
• 3.4. Convexity
• 3.5. Gradient descent and its convergence analysis
• 3.6. Application to logistic regression
• 3.7. Exercises
• 3.8. Online supplementary material
• Data: lebron.csv, SAHeart.csv advertising.csv customer_airline_satisfaction.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 4: Singular value decomposition

• 4.1. Motivating example: visualizing viral genomes
• 4.2. Background: review of matrix rank and spectral decomposition
• 4.3. Approximating subspaces and the SVD
• 4.4. Power iteration
• 4.5. Application: principal components analysis
• 4.6. Further applications of the SVD: low-rank approximations and ridge regression
• 4.7. Exercises
• 4.8. Online supplementary material
• Data: h3n2-snp.csv, h3n2-other.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 5: Spectral graph theory

• 5.1. Motivating example: uncovering social groups
• 5.2. Background: basic concepts in graph theory
• 5.3. Variational characterization of eigenvalues
• 5.4. Spectral properties of the Laplacian matrix
• 5.5. Application: graph partitioning via spectral clustering
• 5.6. Erdős-Rényi random graph and stochastic blockmodel
• 5.7. Exercises
• 5.8. Online supplementary material
• Data: wisconsin-regions.png, two-badgers.jpg, cell-nuclei.png
• Just the code: Notebook
• Other: Auto-quizzes

Chap 6: Probabilistic models: from simple to complex

• 6.1. Motivating example: tracking location
• 6.2. Background: introduction to parametric families and maximum likelihood estimation
• 6.3. Modeling more complex dependencies 1: using conditional independence
• 6.4. Modeling more complex dependencies 2: marginalizing out an unobserved variable
• 6.5. Application: linear-Gaussian models and Kalman filtering
• 6.6. Exercises
• 6.7. Online supplementary material
• Data: twitter-sentiment.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 7: Random walks on graphs and Markov chains

• 7.1. Motivating example: discovering mathematical topics
• 7.2. Background: elements of finite Markov chains
• 7.3. Limit behavior 1: stationary distributions
• 7.4. Limit behavior 2: convergence to equilibrium
• 7.5. Application: random walks on graphs and PageRank
• 7.6. Further applications: Gibbs sampling and generating images
• 7.7. Exercises
• 7.8. Online supplementary material
• Data: mathworld-adjacency.csv, mathworld-titles.csv
• Just the code: Notebook
• Other: Auto-quizzes

Chap 8: Neural networks, backpropagation and stochastic gradient descent

• 8.1. Motivating example: classifying natural images
• 8.2. Background: Jacobian, chain rule, and a brief introduction to automatic differentiation
• 8.3. Building blocks of AI 1: backpropagation
• 8.4. Building blocks of AI 2: stochastic gradient descent
• 8.5. Building blocks of AI 3: neural networks
• 8.6. Exercises
• 8.7. Online supplementary material
• Data: lebron.csv, SAHeart.csv advertising.csv
• Just the code: Notebook
• Other: Auto-quizzes

Programming languages

• Python: I recommend using Google Colaboratory to run the notebooks. Some resources for learning Python:
  • A good place to start is this tutorial.
  • This textbook has many excellent notebooks about the basics of Numpy, Pandas and matplotlib.
• Julia, R, etc.: If you would like to use a different programming language, try converting the code in the notebooks with your favorite AI chatbot.

Additional Reading

The material on this website was partly influenced by the following excellent textbooks.

• [Ara] C. Arangala, Linear Algebra With Machine Learning and Data, Chapman & Hall, 2023
• [Axl] S. Axler, Linear Algebra Done Right, Springer, 2015
• [BHK] A. Blum, J. Hopcroft, R. Kannan, Foundations of Data Science, Cambridge University Press, 2020
• [Bis] C. Bishop, Pattern Recognition and Machine Learning, Springer, 2006 (Chaps 2, 8, 9, 13)
• [Data8] A. Adhikari, J. DeNero, D. Wagner, Computational and Inferential Thinking: The Foundations of Data Science
• [DS100] S. Lau, J. Gonzalez, D. Nolan, Learning Data Science, O'Reilly, 2023
• [ISLP] G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning: with Applications in Python, Springer, 2023
• [MSMB] S. Holmes, W. Huber, Modern Statistics for Modern Biology, Cambridge University Press, 2019
• [Nic] B. Nica, A Brief Introduction to Spectral Graph Theory, EMS Textbooks in Mathematics, 2018
• [Sol] J. Solomon, Numerical algorithms, CRC Press, 2015 (Chaps 4-7)
• [Str] G. Strang, Linear Algebra and Learning from Data, Wellesley-Cambridge Press, 2019
• [TB] L. N. Trefethen, D. Bau, III, Numerical Linear Algebra, SIAM, 1997
• [VMLS] S. Boyd and L. Vandenberghe. Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares. Cambridge University Press, 2018
• [Wri] S. Wright, Optimization Algorithms for Data Analysis, in: The Mathematics of Data, AMS, 2018 (Sections 2-4)
• [WR] S. Wright, B. Recht, Optimization for Data Analysis, Cambridge University Press, 2022