632 Introduction to Stochastic Processes, Spring 2020 Lecture 3

Meetings: TuTh 9:30-10:45 Soc Sci 6102

Instructor: Timo Seppäläinen

Office: 425 Van Vleck. Office Hours: after class, or by appointment.
Phone: 263-3624

E-mail: seppalai and then at math dot wisc dot edu

Course Assistant: Daniel Szabo, office hours 6-8 PM Mondays and 5-8 PM Tuesdays in Sterling B309.

Graders: Yingda Li, yli67 at wisc and then edu. Yuan Ma, ma227 at wisc and then edu.

This is the course homepage. Part of this information is repeated in the course syllabus that you find on Canvas. Here you find our weekly schedule and updates on scheduling matters. Deadlines from the Registrar's page.

About this course

632 is a survey of five important classes of stochastic processes:

discrete-time Markov chains,
martingales,
Poisson processes,
renewal processes,
continuous-time Markov chains.

The material is treated at a level that does not require measure theory. Consequently technical prerequisites for this course are light: calculus, introductory probability and linear algebra are sufficient. However, the material is sophisticated, so a degree of intellectual maturity and a willingness to work hard are required.

Good knowledge of undergraduate probability at the level of UW-Madison Math 431 (or an equivalent course) is required. This means familiarity with basic probability models, random variables and their probability mass functions and distributions, expectations, joint distributions, independence, conditional probabilities and conditional expectations, the law of large numbers and the central limit theorem. Especially the multivariate topics (joint distributions, conditional expectations) are used throughout 632. If you wish to acquire a book for review, the Math 431 textbook Introduction to Probability by Anderson, Seppäläinen and Valkó is recommended.

In class we go through theory, examples to illuminate the theory, and techniques for solving problems. Homework exercises and exam problems are paper-and-pencil calculations with examples and special cases, together with short proofs.

A typical advanced math course follows a strict theorem-proof format. 632 is not of this type. Mathematical theory is discussed in a precise fashion but only some results can be rigorously proved in class. This is a consequence of time limitations and the desire to leave measure theory outside the scope of this course. Interested students can find more proofs in the literature. For a thoroughly rigorous probability course students should sign up for the graduate probability sequence Math/Stat 733-734 which requires a background in measure theory from Math 629 or 721. An undergraduate sequel to 632 in stochastic processes is Math 635 Introduction to Brownian motion and stochastic calculus. Math 635 requires undergraduate analysis Math 521 as background.

Textbook

Lecture notes will be provided on Canvas. The following textbook is used on the side:

Rick Durrett: Essentials of Stochastic Processes, 3rd edition. We expect to cover parts of Chapters 1-5. UW-Madison students can download this textbook for free through SpringerLink.

Other textbooks for supplemental reading:

Geoffrey Grimmett and David Stirzaker: Probability and Random Processes, Oxford University Press
Greg Lawler: Introduction to Stochastic Processes, Chapman and Hall
Sidney Resnick: Adventures in Stochastic Processes, Birkhauser.
Sheldon Ross: Stochastic Processes, Wiley
Sheldon Ross: Introduction to Probability Models, Academic Press

Canvas

Homework assignments, solutions to homework, and lecture notes will be posted on Canvas.

Piazza

We will use Piazza for online class discussion. On Piazza you can post your math questions and answer other students' questions. You can get to Piazza through Canvas.

Evaluation

Course grades will be based on homework (with occasional quizzes possible) (20%), two in-class midterm exams (20%+20%), and a comprehensive final exam (40%). Midterm exams will be on the following dates:

Exam 1 Thursday February 27 (Week 6) in class.
Exam 2 Friday April 10 (Week 11) on Canvas.

If you have to miss a midterm exam, I can increase the weight of the final exam to cover the missed midterm exam. The final exam is comprehensive: it covers the entire course.

Final exam originally scheduled for Friday May 8, 5:05-7:05 PM. But we can decide on our take-home exam time later.

To each exam you are allowed to bring handwritten notes: one 2-sided 8x11 sheet to Exam 1, two sheets to Exam 2, and three sheets to the final exam. But no calculators, cell phones, or other gadgets will be permitted in exams and quizzes, only pencil and paper.

Here are grade lines that can be guaranteed in advance. A percentage score in the indicated range guarantees at least the letter grade next to it.

[100,90] A, (90,87) AB, [87,76) B, [76,74) BC, [74,62) C, [62,50) D, [50,0] F.

Course grades will not be curved, but depending on the outcome of the exams, the tentative grade lines above may be adjusted.

Spring 2020 weekly schedule

Here we record class topics as we cover them.

Week Topics

1 1/21-24 Stochastic processes, state space, finite-dimensional probabilities. IID processes. Strong law of large numbers. Renewal processes and the SLLN for renewal processes.

2 1/27-31 Renewal-reward processes and their SLLN. Markov chains and transition probabilities. Computation with transition probabilities. Simple random walk, gambler's ruin, success runs. Homework 1 due Wednesday 4 pm.

3 2/3-7 Multistep transition probabilities. Markov property into the infinite future. Probability of win in symmetric gambler's ruin. Strong Markov property. Recurrence and transience. Homework 2 due Wednesday 4 pm.

4 2/10-14 Recurrence and transience. Simple random walk. Canonical decomposition of the state space. Absorption probabilities. Homework 3 due Wednesday 4 pm.

5 2/17-21 Absorption probabilities. Invariant distributions. Homework 4 due Friday 4 pm.

6 2/24-28 Review for Exam 1. Invariant distributions. Exam 1 on Thursday in class.

7 3/2-6 Strong law of large numbers for Markov chains. Markov chain convergence theorem.

8 3/9-13 Conditional expectations. Martingales. Homework 5 due Friday 4 pm.

* 3/16-20 SPRING BREAK.

9 3/23-27 Conditional expectations. Martingales. Martingale convergence theorem. Homework 6 due Sunday 4 pm as a PDF file in Canvas.

10 3/30-4/3 Generating functions and the branching process. Homework 7 due Sunday 4 pm as a PDF file in Canvas.

11 4/6-10 Exam 2 on Friday: open-book take-home exam on Canvas.

12 4/13-17 Poisson processes. Homework 8 due Sunday 4 pm as a PDF file in Canvas.

13 4/20-24 Construction of continuous-time Markov chains. The meaning of jump rates. Jump rate as derivative of transition probability.

14 4/27-5/1 Continuous-time Markov chains. Homework 9 due Thursday 4 pm as a PDF file in Canvas.

Week	Topics
1 1/21-24	Stochastic processes, state space, finite-dimensional probabilities. IID processes. Strong law of large numbers. Renewal processes and the SLLN for renewal processes.
2 1/27-31	Renewal-reward processes and their SLLN. Markov chains and transition probabilities. Computation with transition probabilities. Simple random walk, gambler's ruin, success runs. Homework 1 due Wednesday 4 pm.
3 2/3-7	Multistep transition probabilities. Markov property into the infinite future. Probability of win in symmetric gambler's ruin. Strong Markov property. Recurrence and transience. Homework 2 due Wednesday 4 pm.
4 2/10-14	Recurrence and transience. Simple random walk. Canonical decomposition of the state space. Absorption probabilities. Homework 3 due Wednesday 4 pm.
5 2/17-21	Absorption probabilities. Invariant distributions. Homework 4 due Friday 4 pm.
6 2/24-28	Review for Exam 1. Invariant distributions. Exam 1 on Thursday in class.
7 3/2-6	Strong law of large numbers for Markov chains. Markov chain convergence theorem.
8 3/9-13	Conditional expectations. Martingales. Homework 5 due Friday 4 pm.
* 3/16-20	SPRING BREAK.
9 3/23-27	Conditional expectations. Martingales. Martingale convergence theorem. Homework 6 due Sunday 4 pm as a PDF file in Canvas.
10 3/30-4/3	Generating functions and the branching process. Homework 7 due Sunday 4 pm as a PDF file in Canvas.
11 4/6-10	Exam 2 on Friday: open-book take-home exam on Canvas.
12 4/13-17	Poisson processes. Homework 8 due Sunday 4 pm as a PDF file in Canvas.
13 4/20-24	Construction of continuous-time Markov chains. The meaning of jump rates. Jump rate as derivative of transition probability.
14 4/27-5/1	Continuous-time Markov chains. Homework 9 due Thursday 4 pm as a PDF file in Canvas.

Instructions for homework

Homework is collected in class before the due date, or alternately can be brought to the instructor's office or mailbox by the time it is due. No late papers will be accepted. You can bring the homework earlier to the instructor's office or mailbox.
Observe rules of academic integrity. Handing in plagiarized work, whether copied from a fellow student or off the web, is not acceptable. Plagiarism cases will lead to sanctions. You are certainly encouraged to discuss the problems with your fellow students, but in the end you must write up and hand in your own solutions.
Organize your work neatly. Use proper English. Write in complete English or mathematical sentences. Answers should be simplified as much as possible. If the answer is a simple fraction or expression, a decimal answers from a calculator is not necessary. But for some exercises you need a calculator to get the final answer.
As always in mathematics, numerical answers alone carry no credit. It's all in the reasoning you write down.
Put problems in the correct order and staple your pages together. Do not use paper torn out of a binder.
Be neat. There should not be text crossed out. Recopy your problems. Do not hand in your rough draft or first attempt. Papers that are messy, disorganized or unreadable cannot be graded.

Interested in what is happening in probability theory at the research level?

Check out the Probability Seminar for talks on topics that might interest you.

Timo Seppalainen