Plan for upgrading alpha.math.wisc.edu on December 23,2019

Phase 1

1. vonneumann: Copy backup file, Mon.tgz, from bravo. Unzip into /root/tmp/, not /tmp. Make sure backup is complete.
2. vonneumann: Copy bind9 config files from bravo.
3. vonneumann: Restart bind9, test.
4. vonneumann: Copy cert, key, & csr for ldap2.math.wisc.edu from bravo.
5. vonneumann: Change slapd.conf for ldap2 instead of vonneumann,

       sed -i s/vonneumann/ldap2/g slapd.conf
       

6. vonneumann: Run resyncldap. Test.
7. vonneumann: Copy dhcp config, including conf.d, from bravo. Do not restart, testing is not possible. As long as we have config files, we can get this working later.
8. vonneumann: Nothing to do for ntp.
9. vonneumann: Set hostname to bravo,

       echo bravo> /etc/hostname
       

10. vonneumann: Change IP address in /etc/networking/interfaces to 144.92.166.14
11. bravo: Copy backup file, Mon.tgz, from alpha. Unzip into /root/tmp/ (not/tmp). Verify that all files are there.
12. bravo: Copy bind9 config files from alpha:/etc/bind9/. Change config to look for zone files in /etc/bind9/conf-enabled/ instead of /var/cache/bind9/.
13. bravo: Copy zone files from alpha to /etc/bind9/conf-available/. Make sure there are links to all of them in /etc/bind9/conf-enabled/.

        for File in ../conf-available/*; do ln -s "$File"; done
        

    Make some trivial change that can be checked for. Possible change cname for dbserver2.
14. bravo: Restart bind9. Test. Bravo should think its master. May get errors about not being able to talk to slave.
15. bravo: Copy cert, key, and csr for ldap1.math.wisc.edu from alpha.
16. bravo: Change slapd.conf from ldap2 to ldap1,

        sed -i s/ldap2/ldap1/g slapd.conf".
        

17. bravo: Restart slapd, test.

        ldapsearch -x -LLL -ZZ  -H ldap://ldap1.math.wisc.edu -D "uid=nemo,ou=people,dc=math,dc=wisc,dc=edu" -W "uid=nemo"
        

18. bravo: Copy all dhcp config from alpha, including conf.d. Do not restart, testing is not possible.
19. bravo: Nothing to do for ntp.
20. bravo: Change cname for dbserver1 to alpha. This may be a goodtrivial change to make for testing purposes as mentioned above.
21. bravo: Change hostname to alpha,

        echo alpha > /etc/hostname
        

22. bravo: Change IP address in /etc/networking/interfaces from 144.92.166.14 to 144.92.166.12
23. alpha: Disable bind9,

        systemctl disable bind9
        

24. alpha: Disable dhcp. Systemd will not work. Suggest renaming dhcpd.conf.
25. alpha: change hostname to vonneumann, echo vonneumann > /etc/hostname
26. alpha: Change IP address in /etc/networking/interfaces to 144.92.166.23.
27. Reboot all 3 servers.

Phase 2

Note: The names below represent the new host names. Alpha is the old bravo, bravo is the old vonneumann, and vonneumann is the old alpha.

1. hilbert: Clear dns cache

       systemd-resolve --flush-caches
       

2. Make sure web site comes back up.
3. vonneumann: Make sure dns & dhcp are not running. Optionally stop slapd and ntp.
4. ping 144.92.166.12
5. host vv101c 144.92.166.12
6. host vv101d alpha
7. ldapsearch -x -ZZ -H ldap://ldap1.math.wisc.edu uid=nemo
8. ssh nemo@penguin
9. dhclient -v eno1
10. Reboot penguin, rerun above tests.
11. SSh to lambeau, rerun above tests.
12. Call DoIT hostmaster, make sure it looks good to him.

Phase 3

1. alpha: Chec syslog for slapd startup errors.
2. alpha: Check syslog for dhcp sync errors.
3. alpha:
4. Check syslog for ntp offset errors.
5. Make trivial change to dns config. Restart bind9. Make sure change is replicated on bravo.
6. Make trivial change to dhcp config. Make sure change is replicated on bravo.
7. Change a password. Make sure works.
8. Reboot penguin again. Make sure ntp servers are correct.
9. Check hosts file on bravo. May have put stuff for testing in there.
10. Configure ansible on alpha to update dhcp config files on bravo.
11. Configure bravo as replica mysql server.

Copyright 2019, The Board Of Regents, University Of Wisconsin-Madison