Handouts:
Final Exam: Thursday December 18, 12:25 PM, Soc Sci 6102. Duration: 2 hours. Topics covered: entire course. You can expect that all 4 models (discrete-time Markov chains, Poisson point processes, continuous-time Markov chains, and renewal processes) appear in the exam. Pencil, paper, and your three sheets of notes allowed in the exam. No calculators, cell phones, or other gadgets permitted. Please bring your own paper to the exam, but no spiral bound notebooks, just loose paper. No take-home part in the final exam.
HOMEWORK ASSIGNMENTS. (Numbers refer to exercises in Resnick's book.)
| Homework 1, due September 9. Exercises 1.3, 1.12, 1.15. For 1.3 you can assume that the gf's are differentiable at s=1. In 1.12 replace the word 'recursion' with the word 'equation.' As a hint, you might notice that each offspring of the progenitor serves in turn as a progenitor of a new process. For the mean of S find a general formula in terms of m and apply it to the special cases. |
| Homework 2. Due September 23. Problem 1 and from the book Exercises 2.2, 2.4, 2.5. In 2.2 it may be simplest to first figure out what the transition matrix must be and then verify that the process is a Markov chain with this transition matrix. In 2.4 note that under the probability measure Pi the Markov chain starts with X0=i. In 2.5 produce an example of a Markov chain and a function f on the state space and show that your example fails the Markov property. |
| Homework 3, due October 7. Exercises 2.12, 2.30, 2.62, 2.69(a)(b). In 2.30 find all the invariant distributions. Do not miss the helpful formula in Proposition 2.12.3. For 2.62 find the Moran storage model in Section 2.2. Notice the mistake: it should read p+q+r=1. Please no answers without justification. This is the last homework before Exam 1. |
| Homework 4, due October 30. Exercises 3.1, 3.20 and 3.45. In 3.1 give the answers in terms of the parameters p and μi. For 3.20 add this part (d): find the limit in (c) explicitly for the case where F1 and F2 are exponential distributions with distinct parameters α and β. In 3.45 do only parts (a), (b)(1) and (b)(2). In (b) assume the renewal process is pure (maybe that is what the author means by "ordinary"). |
| Homework 5, due November 11. Exercises 4.4, 4.5, 4.9. In 4.5 you can use marking, thinning, or do a simple computation. In 4.9 first compute the Laplace transform of the Exp(α) distribution. Probability distributions of nonnegative random variables are determined by their Laplace transforms, this is stated on p. 181. This is the last homework before Exam 2. |
| Homework 6 (last one), due December 9. Exercises 5.10, 5.14, 5.22. In 5.10, instead of the numbers given in the exercise, assume customers arrive at rate a and servers serve at rate b. Then in part (b) let the single server serve at rate 2b. (This is to avoid the structure of the problem getting lost in the arithmetic with numbers.) "Capacity of at most 3 customers" means that when the system has 3 customers present, all arriving customers turn away and are lost. In case you argue via long-term probabilities of the Markov chain, give a justification for why this is correct. This same question is posed also in 5.22. |
BONUS PROBLEMS for those ready for some extra challenge. Hand in bonus problems separately from regular homework. They are due any time before December.
| Problem 1. Derive the book's formula (1.6.4) for random walk from Exercise 1.12 by creating a connection between the random walk and a certain branching process so that hitting the point 1 corresponds to extinction. Hint: you can construct the random walk from segments of path between successive +1 steps. |
| Problem 2. Let S0, S1, S2,... be a simple symmetric random walk started at S0=0. Let Yn= max{S0,S1,...,Sn} be the running maximum of the random walk. Determine whether Yn is a Markov chain. (In other words, prove that Yn satisfies the Markov property, or prove that the Markov property fails.) |
| Problem 3. Exercise 2.25. |
| Problem 4. Exercise 4.42. You might think of the customers that arrive during one cycle of the bus renewal process. Use what we know about the limit of the forward recurrence time. |
| Week | Class period 1 | Class Period 2 |
|---|---|---|
| 1 | 1.1-1.3 Generalities about probability spaces and random variables. Convolution. Generating functions. | 1.3-1.4 More about generating functions. The simple branching process. |
| 2 | 1.4 Characterization of extinction probability. Brief discussion of 1.5-1.7. | 1.8 Stopping times and Wald's identity. 2.1-2.2 Beginning of Markov chains. |
| 3 | 2.1-2.3 Markov property. Examples. Construction of Markov chains. Higher order transition probabilities. | 2.4 Decomposition of the state space. 2.5 Start strong Markov property. |
| 4 | 2.5 Strong Markov property and the dissection principle. 2.6 Recurrence and transience. | 2.6 continued. Examples. 2.7 Periodicity. 2.8 Class properties. |
| 5 | 2.10 Canonical decomposition. 2.12 Invariant distributions. | 2.12 Invariant distributions continued. |
| 6 | 2.12.1 and 2.13 Long term behavior: strong law of large numbers and the convergence theorem. | Conclusion of Markov chains: doubly stochastic and reversible chains, absorption probabilities. |
| 7 | Exam I | 3.1-3.3. Begin renewal processes. Quick introduction to Riemann-Stieltjes integrals and convolutions. Renewal function. Exponential example. |
| 8 | First limit theorems. Begin renewal equation. Forward recurrence time as example. | 3.5 Renewal equation. 3.7 Blackwell's renewal theorem. Key renewal theorem. |
| 9 | 3.7 Smith's theorem. Conclusion of renewal theory. | Ch 4. Begin point processes. |
| 10 | Discussion of renewal theory homework. 4.4 Thinning of Poisson processes. | 4.4 Superposition and marking of Poisson processes. |
| 11 | 4.5 Conditioning a Poisson process on the number of points in a set. Order statistic property. | Laplace functional. Construction of the general Poisson point process. |
| 12 | Exam II | 5. Begin continuous time Markov chains: Markov property, construction, the possibility of explosion. |
| 13 | 5.4 Backward equation, generator matrix. | Thanksgiving. |
| 14 | Invariant and reversible distributions. Limit theorems. M/M/1 and M/M/s queues. | Regeneration. Uniformizable chains. Examples. |
| 15 | Transition probability for uniformizable chain. Invariant distributions for continuous time chain and its jump chain. Reversibility for queues. | Discussion about the last homework and further probability courses. Brief intoduction to martingales. |
632 is a survey of some important classes of stochastic processes: Markov chains in both discrete and continuous time, point processes, and renewal processes. The material is treated at a level that does not require measure theory. Consequently technical prerequisites needed for this course are light: calculus and linear algebra are sufficient. However, the material is sophisticated, so a degree of intellectual maturity and a willingness to work hard are required. For this reason some 500-level work in mathematics is recommended for background, preferably in analysis (521).
Good knowledge of undergraduate probability at the level of UW-Madison Math 431 (or an equivalent course) is required. This means familiarity with basic probability models, random variables and their probability mass functions and distributions, expectations, joint distributions, independence, conditional probabilities, the law of large numbers and the central limit theorem. The handout Probability Basics covers some of these prerequisites. If you need a thorough review of basics, the textbook A First Course in Probability by S. Ross is recommended.
In class we go through theory, examples to illuminate the theory, and techniques for solving problems. Homework exercises and exam problems are paper-and-pencil calculations with examples and special cases.
A typical advanced math course follows a strict theorem-proof format. 632 is not of this type. Mathematical theory is discussed in a precise fashion but only some results can be rigorously proved in class. This is a consequence of time limitations and the desire to leave measure theory outside the scope of this course. Interested students can find the proofs in the textbook. For a thoroughly rigorous probability course students should sign up for the graduate probability sequence 831-832.
Check out the Probability Seminar for talks on topics that might interest you.