Skip to main content

Disaster Recovery Checklist

# Global Architecture and Request Flow

Purpose

##This Purposechecklist provides a concise response plan for major Bytek homelab failures.

The Bytekobjective homelabis providesto privatelyrestore hostedcore identity,infrastructure collaboration,before documentation, monitoring, and project-managementapplication services.

TheAvoid environmentmaking several unrelated configuration changes simultaneously.

Initial Assessment

Before changing configuration:

    Determine whether the problem affects one service or the entire environment. Check electrical power. Check the Proxmox host. Check the home network. Check internet connectivity. Check Pi-hole. Check WireGuard. Check Traefik. Check Authentik. Check application VMs. Review Uptime Kuma. Record the observed symptoms.

    Do not rely only on browser errors.

    Test direct IPs and backend ports where possible.

    Complete Proxmox Host Failure

      Confirm server power. Check the physical console. Confirm the firmware detects all storage devices. Confirm the Proxmox system NVMe. Confirm the user-data NVMe. Confirm the backup HDD. Attempt normal Proxmox boot. Review boot messages. Confirm network bridge configuration. Confirm pveproxy. Confirm TCP 8006. Use root@pam.

      If the system NVMe failed:

        Replace the failed system storage. Install Proxmox. Restore network configuration. Restore storage configuration. Reconnect user-data. Remount pve-backup. Restore VMs and LXCs. Verify local authentication. Restore the Authentik Proxmox realm. Test OIDC only after local access works.

        Pi-hole Failure

          Open Proxmox using 192.168.2.254. Confirm the Pi-hole LXC is hostedrunning. Open the LXC console. Confirm address 192.168.2.65. Confirm TCP port 53. Confirm UDP port 53. Confirm local DNS records. Confirm public upstream resolution. Review Pi-hole logs. Restore from backup if necessary.

          During the outage, access services using private IP addresses where supported.

          Do not make Pi-hole recovery depend on Proxmox VE and follows these design principles:Authentik.

          -

          One

          WireGuard majorFailure

          service
          per
          VMConfirm orthe LXC.
          -WireGuard Publicgateway servicesguest. enterConfirm throughaddress a192.168.2.64. Confirm the WireGuard interface. Confirm the VPS insteadpeer. ofConfirm directthe routerlatest porthandshake. Confirm transfer counters. Confirm IPv4 forwarding.
          - WireGuardConfirm transportsfirewall publicrules. trafficConfirm securelyNAT rules. Test Traefik connectivity from the tunnel. Test a public hostname externally.

          If the keys are lost or compromised, rotate the affected peer key pair and update the opposite peer.

          Public VPS Failure

            Open the provider console. Confirm the VPS is running. Confirm public IP 38.29.213.101. Confirm SSH service. Confirm firewall rules. Confirm WireGuard. Confirm TCP 443. Confirm forwarding to the home network.
            -tunnel. TraefikConfirm terminatesdisk HTTPSspace. andReview routessystem requestslogs. by hostname.
            -
            Authentik provides centralized authentication, MFA, and

            LAN access policies.
            -should continue through Pi-hole providesand LANTraefik DNS,while splitthe DNS,VPS is unavailable.

            Traefik Failure

              Confirm the Traefik VM owns 192.168.2.182. Confirm Docker. Confirm the Traefik container. Confirm TCP 443. Confirm container DNS. Review Traefik logs. Check recently changed dynamic YAML. Search for tabs. Test the direct application backend. Restore the previous dynamic file if necessary. Confirm certificate state. Do not delete acme.json.

              If all applications fail simultaneously but direct backends work, prioritize Traefik, Pi-hole, and DHCPcertificates.

              reservations.
              -
              Data-bearing

              Authentik applicationsFailure

              use
                Sign into Proxmox using root@pam. Confirm the Authentik VM owns 192.168.2.162. Confirm Docker. Confirm PostgreSQL. Confirm Redis. Confirm Authentik server. Confirm Authentik worker. Test direct readiness. Test routed readiness. Review Authentik logs. Review Traefik. Restore Authentik from backup if necessary.

                Use application break-glass access during recovery:

                Application Recovery Method Proxmox root@pam Nextcloud Local administrator BookStack Switch to standard authentication Uptime Kuma Restore local authentication Traefik Basic Auth or restore known-good route

                Nextcloud Failure

                  Confirm VM address 192.168.2.100. Confirm the 500 GB data disk. Confirm /mnt/nextcloud-data. Confirm free disk space. Confirm Docker. Open AIO management by private IP. Confirm PostgreSQL. Confirm Redis. Confirm Nextcloud. Confirm Apache. Confirm TCP 11000. Confirm Traefik. Use the local administrator if OIDC fails. Restore both virtual disks if recovery is required.

                  Do not start normal Nextcloud operation if the data mount is missing.

                  Vikunja Failure

                    Confirm VM address 192.168.2.121. Confirm Docker. Confirm PostgreSQL. Confirm the Vikunja container. Confirm TCP 3456. Test /health. Run the Vikunja doctor command. Check attachment permissions. Check Traefik. Check Authentik after application health is restored.

                    BookStack Failure

                      Confirm the BookStack VM. Confirm the current private IP. Confirm Docker. Confirm MariaDB. Confirm the BookStack container. Confirm TCP 6875. Review the Laravel log. Confirm APP_URL. Confirm the application key. Confirm Traefik. Switch to standard authentication if OIDC blocks access. Restore the VM if application data is damaged.

                      The application key and database must be restored together.

                      Uptime Kuma Failure

                        Confirm VM address 192.168.2.115. Confirm Docker. Confirm the Uptime Kuma container. Confirm persistent data. Confirm embedded database health. Confirm container DNS. Test TCP 3001. Confirm Traefik. Confirm Authentik Proxy Provider. Restore local authentication if proxy access fails.

                        The absence of monitoring does not necessarily mean every monitored system is down.

                        Backup HDD Failure

                          Confirm /dev/sda. Confirm /dev/sda1. Confirm filesystem UUID. Confirm /etc/fstab. Confirm /mnt/pve/pve-backup. Confirm Proxmox storage state. Review SMART health. Stop scheduled backups if the filesystem is unsafe. Replace the backup HDD if necessary. Create a separatefresh LVM-thinbackup storageset pool.
                          -immediately Proxmoxafter replacement.

                          Do not allow backups areto write into an unmounted directory on the root filesystem.

                          User-Data NVMe Failure

                          Expected impact includes Nextcloud user data and any other data disks stored on user-data.

                          Response:

                            Stop affected guests. Avoid further writes. Confirm the physical NVMe. Review NVMe health. Confirm the lvm1 volume group. Confirm the data thin pool. Identify the latest valid backups. Replace the failed storage if required. Recreate the user-data storage. Restore affected data disks. Validate guests offline. Return services to production.

                            Authentication Lockout

                            Proxmox

                            Use root@pam.

                            Nextcloud

                            Use the local administrator and direct local-login path.

                            BookStack

                            Set AUTH_METHOD to standard, recreate the BookStack container, and use the local administrator.

                            Uptime Kuma

                            Restore the direct Traefik route, allow temporary private access, and re-enable local authentication.

                            Traefik Dashboard

                            Restore the previous dashboard YAML or use retained Basic Auth.

                            Authentik

                            Use the local Authentik administrator or restore Authentik from backup.

                            Restore Validation

                            Before replacing production with a dedicatedrestored guest:

                              Restore under a temporary guest ID. Disconnect the network adapter. Start through the Proxmox console. Confirm operating-system boot. Confirm all intended disks. Confirm mount points. Confirm Docker. Confirm database. Confirm application data. Confirm configuration and secrets. Shut down the test clone. Schedule the production cutover. Shut down the failed production guest. Assign the expected network identity to the restored guest. Start the restored guest. Test direct backend access. Test Traefik. Test Authentik. Test external access. Update documentation.

                              Emergency Credentials

                              The password manager must contain:

                                Proxmox root@pam password. Authentik administrator credentials. Nextcloud local administrator credentials. BookStack break-glass administrator credentials. Uptime Kuma administrator credentials. Traefik Basic Auth credentials. VPS SSH private key. Home-infrastructure SSH keys. WireGuard configuration and recovery keys. WHC cPanel credentials. SMTP credentials. Database recovery credentials where required.

                                Do not store the actual credential values in BookStack.

                                BookStack should record only the credential purpose and storage location.

                                Recovery Priority

                                Restore services in this order:

                                  Proxmox VE. Storage pools. Pi-hole. WireGuard Gateway. Authentik database and Authentik. Traefik. Uptime Kuma. Nextcloud. Vikunja. BookStack. Future media and game services.

                                  Incident Record

                                  After a significant incident, record:

                                    Date and time. Affected services. Initial symptoms. Root cause. Recovery actions. Backup used. Data loss, if any. Configuration changes. Monitoring gaps. Preventive actions. Documentation changes.

                                    Do not include passwords, tokens, private keys, or session cookies.

                                    Final Recovery Checklist

                                      Proxmox management works. Required storage is active. Backup HDD is mounted. Pi-hole resolves internal HDD.
                                      -and Managementpublic interfacesnames. remainWireGuard LANhandshake orexists. VPNVPS onlyingress wheneverworks. practical.
                                      -Authentik Eachreadiness criticalreturns serviceHTTP retains200. anTraefik independentroutes load. Certificates are valid. Uptime Kuma is healthy. Nextcloud data disk is mounted. Nextcloud file access works. Vikunja health returns HTTP 200. BookStack documentation opens. OIDC works. Local break-glass accounts work. Public access works externally. New backup completes successfully. Documentation is updated.

                                      Document Control

                                        Owner: Bryan Gagne-Plante Primary recovery method.platform:

                                        ---

                                        Proxmox

                                        ##VE

                                        High-LevelPrimary Architecturebackup

                                        ```text
                                        Internetstorage:

                                        users
                                        pve-backup Backup |
                                        mount:                                         /mnt/pve/pve-backup v
                                        WHCOffsite publicbackup: DNS
                                        Not configured |
                                        Last verified: v
                                        PublicYYYY-MM-DD VPS
                                        38.29.213.101
                                        Last disaster-recovery |
                                        exercise:                                         YYYY-MM-DD |Last WireGuardfull tunnel
                                        restore test:                                         v
                                        HomeYYYY-MM-DD WireGuardLast gateway
                                        192.168.2.64
                                        authentication-recovery test:                                         |
                                        YYYY-MM-DD Last v
                                        Traefik
                                        192.168.2.182
                                        backup-HDD review:                                         |
                                        YYYY-MM-DD Known +-->limitations: Authentik
                                        One Proxmox |host, 192.168.2.162:9000
                                        one local |
                                        backup disk, +-->and Nextcloudno AIO
                                        offsite recovery |copy 192.168.2.100:11000
                                            |
                                              +--> Uptime Kuma
                                              |    192.168.2.115:3001
                                              |
                                              +--> Vikunja
                                              |    192.168.2.121:3456
                                              |
                                              +--> BookStack
                                                   <BOOKSTACK_IP>:6875
                                        Back to top