632 Introduction to Stochastic Processes, Lecture 3

Fall 2018

Meetings: TuTh 9:30-10:45
Instructor: Timo Seppäläinen
Office: 425 Van Vleck. Office Hours: after class, or by appointment.
Phone: 263-3624
E-mail: seppalai and then at math dot wisc dot edu
Teaching Assistant: Congfang Huang, office hours 1-4 PM Mondays Van Vleck B205.

This is the course homepage. Part of this information is repeated in the course syllabus that you find on Canvas. Here you find our weekly schedule and updates on scheduling matters. The Mathematics Department has also a general information page on this course. Deadlines from the Registrar's page.

About this course

632 is a survey of several important classes of stochastic processes: Markov chains in both discrete and continuous time, point processes, and renewal processes. The material is treated at a level that does not require measure theory. Consequently technical prerequisites for this course are light: calculus and linear algebra are sufficient. However, the material is sophisticated, so a degree of intellectual maturity and a willingness to work hard are required. For this reason some 500-level work in mathematics is recommended for background, preferably in analysis (521).

Good knowledge of undergraduate probability at the level of UW-Madison Math 431 (or an equivalent course) is required. This means familiarity with basic probability models, random variables and their probability mass functions and distributions, expectations, joint distributions, independence, conditional probabilities, the law of large numbers and the central limit theorem. If you wish to acquire a book for review, the Math 431 textbook Introduction to Probability by Anderson, Seppäläinen and Valkó is recommended.

In class we go through theory, examples to illuminate the theory, and techniques for solving problems. Homework exercises and exam problems are paper-and-pencil calculations with examples and special cases, together with short proofs.

A typical advanced math course follows a strict theorem-proof format. 632 is not of this type. Mathematical theory is discussed in a precise fashion but only some results can be rigorously proved in class. This is a consequence of time limitations and the desire to leave measure theory outside the scope of this course. Interested students can find the proofs in the literature. For a thoroughly rigorous probability course students should sign up for the graduate probability sequence Math/Stat 733-734 which requires a background in measure theory from Math 629 or 721. An undergraduate sequel to 632 in stochastic processes is Math 635 Introduction to Brownian motion and stochastic calculus.


Rick Durrett: Essentials of Stochastic Processes. Both 2nd and 3rd edition will work. Homework will be assigned from the December 2011 version of the 2nd edition available from Rick's homepage at Duke University.

We will cover Chapters 1-4, and if time permits, part of Chapter 5.


Homework assignments, solutions to homework, and lecture notes will be posted on Canvas at Learn@UW.


We will use Piazza for online class discussion. On Piazza you can post your math questions and answer other students' questions. You can get to Piazza through Canvas. Our class Piazza page is at piazza.com/wisc/fall2018/fa18math632003/home


Course grades will be based on homework (with occasional quizzes possible) (20%), two midterm exams (20%+20%), and a comprehensive final exam (40%). Midterm exams will be in class on the following dates:

If you have to miss a midterm exam, I will increase the weight of the final exam to cover the missed midterm exam. The final exam is comprehensive: it covers the entire course. To each exam you are allowed to bring handwritten notes: one 8x11 sheet to Exam 1, two sheets to Exam 2, and three sheets to the final exam. But no calculators, cell phones, or other gadgets will be permitted in exams and quizzes, only pencil and paper.

Here are grade lines that can be guaranteed in advance. A percentage score in the indicated range guarantees at least the letter grade next to it.

[100,90] A,   (90,87) AB,  [87,76) B,  [76,74) BC,  [74,62) C,  [62,50) D,  [50,0] F.

Course grades will not be curved, but depending the outcome of the exams the tentative grade lines above may be adjusted.

Instructions for homework

Fall 2018 weekly schedule

Here we record class topics as we cover them. Section numbers refer to Durrett's book.

Week Tuesday Thursday
1 9/5-7 BEFORE SEMESTER Lecture notes on Canvas: IID processes and Markov chains. Simple random walk. Transition probability.
2 9/10-14 Multistep transition probabilities. Recurrence, transience, strong Markov property.
3 9/17-21 Strong Markov property revisited from lecture notes on Canvas. Durrett's book, Section 1.3: Lemma 1.4, Theorem 1.5, definitions of closed and irreducible sets.
Homework 1 due.
Section 1.3 from Durrett finished.
4 9/24-28 Examples of canonical decomposition and recurrence/transience (gambler's ruin, simple random walk). 1.4 Stationary distributions. Examples: two-state MC, gambler's ruin. 1.4 Stationary distributions: renewal chain example (Durrett 1.22). Summary of facts concerning invariant measures and distributions. 1.5 Begin development towards MC convergence theorem: period of a recurrent state.
Homework 2 due.
5 10/1-5 Discussion and examples of the Markov chain convergence theorem: renewal chain, two-state Markov chain, convergence to a stationary stochastic process. Beginning the proof of the convergence theorem. Conclusion of the proof of the Markov chain convergence theorem. Review strong law of large numbers (SLLN) for IID random variables.
6 10/8-12 Dissection principle for Markov chains. Limiting frequency of visits to a state. SLLN for Markov chains.
Homework 3 due.
Exam 1. Covers Sections 1.1-1.5 from Durrett's book.
7 10/15-19 1.6 Detailed balance and reversible Markov chains. 1.8 Exit distributions. 1.9 Exit times.
8 10/22-26 2.1 Exponential, gamma and Poisson distributions.
Homework 4 due.
2.1 Exponential, gamma and Poisson distributions. 2.2. Poisson process on the line.
9 10/29-11/2 2.2 Properties of the homogeneous Poisson process on the line. Exercise 2.22. Poisson process as a limit from a sequence of independent trials. 2.3 Compound Poisson processes. 2.4 Thinning.
Homework 5 due.
10 11/5-9 2.4 Thinning, superposition and conditioning. Conditioning: derivation of the joint conditional density f(x,y)=2/t^2 of (T1,T2) conditional on N(t)=2. Examples.
11 11/12-16 Review of Poisson processes.
Homework 6 due.
Exam 2. Poisson processes.
12 11/19-21 3.1 Laws of large numbers for renewal processes and renewal-reward processes. 3.3 Definition of age and residual lifetime. Stopping times for IID sequences. THANKSGIVING
13 11/26-30 3.3 Limit distributions for age and residual lifetime. Stationary renewal process. 4.1 Continuous-time Markov chains: definitions and first examples.
14 12/3-7 4.1 Construction of continuous-time Markov chains from given rates. 4.2 Kolmogorov's forward and backward equations. Generator matrix. 4.3 Invariant distributions and the convergence theorem.
Homework 7 due.
4.2 Exponentials of matrices. 4.3 Detailed balance and reversibility. Queueing examples.
15 12/10-12 Introduction to Brownian motion. Review questions.
Homework 8 due.

Check out the Probability Seminar for talks on topics that might interest you.

Timo Seppalainen