• [x] Prep

    • [x] Acquire new server
      • [x] Install & configure OS (Alpine Linux)
      • [x] Configure external networking
      • [x] Configure Docker Swarm
      • [x] Configure management backplane
      • [x] Transfer Docker Swarm data
        • [x] Configs
        • [x] Secrets
        • [x] Stacks
    • [x] Sync CDN content
    • [x] Add DNS records for new environment

  • [x] Action

    • [x] Bring down old environment
    • [x] Update DNS records to point to new environment
    • [x] Configure HAProxy on old environment to forward traffic to new environment
    • [x] Migrate volumes
      • [x] DB - Live
      • [x] DB - Replica
      • [x] Redis
    • [x] Bring up services
      • [x] DB
      • [x] Redis
      • [x] Streaming API
      • [x] Redis (ephemeral cache instance)
    • [x] Sync CDN - Final
    • [x] Bring up services (cont.)
      • [x] CDN
      • [x] Web
      • [x] Sidekiq
        • [x] Default
        • [x] Mail
        • [x] Pull
        • [x] Push
        • [x] Scheduled
      • [x] Maintenance
        • [x] Cards
        • [x] Media
    • [x] Migrate volumes (cont.)
      • [x] DB backup configuration
      • [x] DB backup generation working-area
    • [x] Bring up services (cont.)
      • [x] DB backups (schema export, DB export, archiver)
      • [x] DB offsite sync
      • [x] DB stale data pruner
    • [x] Final smoke-test
    • [x] Announce completion

  • [x] Post

    • [x] Monitor traffic on old environment until none remains
    • [x] Shut down docker hosts
    • [x] Take off-site backup of VM disks
    • [x] Secure-erase VM disk files
    • [x] Move other infrastructure off
      • [x] ns2.maff.scot
      • [x] bark.pup.cloud
      • [x] commit.pup.cloud
      • [x] InfluxDB
      • [x] Invidious
      • [x] IRC client
      • [x] Node-RED
        • [x] Instance A
        • [x] Instance B
    • [x] Shut down those docker hosts
    • [x] Shut down NFS server
    • [x] Shut down firewall VM
    • [x] Take off-site backup of firewall and NFS volumes
    • [x] Secure-erase those VM disks
    • [x] Take off-site backup of hypervisor data
    • [x] Remove temporary firewall rule for old environment’s management IP which granted access to a migration service on the new environment
    • [x] Reboot into a netboot recovery environment
    • [x] Secure-erase both HDDs
    • [x] Cancel additional IPs
    • [x] Cancel server
    • [x] Remove stale DNS entries

    Hitches

    • [x] Fix missing mail config in sk_default - caused transactional mail outage post-3.4.4 upgrade
      • [x] Add SMTP_ environment variables to sidekiq default queue runner temporarily
      • [x] Restore commit.pup.cloud
      • [x] Push fix to libloader to add enrich mail to role sidekiq sub-role main
      • [x] Rebuild docker image maffsie/qp-mastodon:v3.4.4 and :latest
        • [x] Remove direct ENV declarations for docker service mastodon_sk_default