# Disaster Recovery Checklist

## Purpose

This checklist provides a concise response plan for major Bytek homelab failures.

The objective is to restore core infrastructure before application services.

Avoid making several unrelated configuration changes simultaneously.

<div id="bkmrk-" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Initial Assessment

Before changing configuration:

<div id="bkmrk-determine-whether-th" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Determine whether the problem affects one service or the entire environment.
2. Check electrical power.
3. Check the Proxmox host.
4. Check the home network.
5. Check internet connectivity.
6. Check Pi-hole.
7. Check WireGuard.
8. Check Traefik.
9. Check Authentik.
10. Check application VMs.
11. Review Uptime Kuma.
12. Record the observed symptoms.

</div>Do not rely only on browser errors.

Test direct IPs and backend ports where possible.

<div id="bkmrk--1" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Complete Proxmox Host Failure

<div id="bkmrk-confirm-server-power" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm server power.
2. Check the physical console.
3. Confirm the firmware detects all storage devices.
4. Confirm the Proxmox system NVMe.
5. Confirm the user-data NVMe.
6. Confirm the backup HDD.
7. Attempt normal Proxmox boot.
8. Review boot messages.
9. Confirm network bridge configuration.
10. Confirm `pveproxy`.
11. Confirm TCP 8006.
12. Use `root@pam`.

</div>If the system NVMe failed:

<div id="bkmrk-replace-the-failed-s" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Replace the failed system storage.
2. Install Proxmox.
3. Restore network configuration.
4. Restore storage configuration.
5. Reconnect `user-data`.
6. Remount `pve-backup`.
7. Restore VMs and LXCs.
8. Verify local authentication.
9. Restore the Authentik Proxmox realm.
10. Test OIDC only after local access works.

---

</div>## Pi-hole Failure

<div id="bkmrk-open-proxmox-using-1" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Open Proxmox using `192.168.2.254`.
2. Confirm the Pi-hole LXC is running.
3. Open the LXC console.
4. Confirm address `192.168.2.65`.
5. Confirm TCP port 53.
6. Confirm UDP port 53.
7. Confirm local DNS records.
8. Confirm public upstream resolution.
9. Review Pi-hole logs.
10. Restore from backup if necessary.

</div>During the outage, access services using private IP addresses where supported.

Do not make Pi-hole recovery depend on Authentik.

<div id="bkmrk--2" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## WireGuard Failure

<div id="bkmrk-confirm-the-wireguar" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm the WireGuard gateway guest.
2. Confirm address `192.168.2.64`.
3. Confirm the WireGuard interface.
4. Confirm the VPS peer.
5. Confirm the latest handshake.
6. Confirm transfer counters.
7. Confirm IPv4 forwarding.
8. Confirm firewall rules.
9. Confirm NAT rules.
10. Test Traefik connectivity from the tunnel.
11. Test a public hostname externally.

</div>If the keys are lost or compromised, rotate the affected peer key pair and update the opposite peer.

<div id="bkmrk--3" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Public VPS Failure

<div id="bkmrk-open-the-provider-co" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Open the provider console.
2. Confirm the VPS is running.
3. Confirm public IP `38.29.213.101`.
4. Confirm SSH service.
5. Confirm firewall rules.
6. Confirm WireGuard.
7. Confirm TCP 443.
8. Confirm forwarding to the home tunnel.
9. Confirm disk space.
10. Review system logs.

</div>LAN access should continue through Pi-hole and Traefik while the VPS is unavailable.

<div id="bkmrk--4" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Traefik Failure

<div id="bkmrk-confirm-the-traefik-" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm the Traefik VM owns `192.168.2.182`.
2. Confirm Docker.
3. Confirm the Traefik container.
4. Confirm TCP 443.
5. Confirm container DNS.
6. Review Traefik logs.
7. Check recently changed dynamic YAML.
8. Search for tabs.
9. Test the direct application backend.
10. Restore the previous dynamic file if necessary.
11. Confirm certificate state.
12. Do not delete `acme.json`.

</div>If all applications fail simultaneously but direct backends work, prioritize Traefik, Pi-hole, and certificates.

<div id="bkmrk--5" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Authentik Failure

<div id="bkmrk-sign-into-proxmox-us" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Sign into Proxmox using `root@pam`.
2. Confirm the Authentik VM owns `192.168.2.162`.
3. Confirm Docker.
4. Confirm PostgreSQL.
5. Confirm Redis.
6. Confirm Authentik server.
7. Confirm Authentik worker.
8. Test direct readiness.
9. Test routed readiness.
10. Review Authentik logs.
11. Review Traefik.
12. Restore Authentik from backup if necessary.

</div>Use application break-glass access during recovery:

<div id="bkmrk-application-recovery" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;"><div aria-expanded="false" class="___5wvz9a0 ftgm304 f1oy3dpc fm6nont f48hbct" data-stable-ignore="true" data-tabster="{"restorer":{"type":1}}" role="presentation" tabindex="0"><div class="___1dmoc29 f10pi13n ftgm304 f1enuhaj fdclmfp f1nbblvp fat0sn4 f1ov4xf1 fekwl8i f1lmfglv f1oz7aqm f1abmfm4 f1w619qj f16h0jq8"><table class="___1vyiefv f1ddd56o f16vktn6 f1ahpp82 f11qra4b f1uinfot fibjyge fvueend f9yszdx f1fu4s3n f3l3pb3 f10ghnd0 f8fmt76 fjvbh62 f1qrqxae f1vw5qpk fc02sbz fxawf59 fymf513 f1aoyrul f1el8yx3 f1pymoxg f1ofu761 fe6itr f7coize f1794535 f1o0pw0q fbjjl9v fk1v6el f16pyhcb f1ixlhx9 f12zef0i flu5r5u f19haqzy f1owmcxx f1oddm8q f1004tna fcoaxci fh0ee9u f15v23i2 f1dmj53 f1r1gcv9 f14z1veh ffufd3x f1ypplot f1660cg"><tbody><tr><th>Application</th><th>Recovery Method</th></tr><tr><td>Proxmox</td><td>`root@pam`</td></tr><tr><td>Nextcloud</td><td>Local administrator</td></tr><tr><td>BookStack</td><td>Switch to standard authentication</td></tr><tr><td>Uptime Kuma</td><td>Restore local authentication</td></tr><tr><td>Traefik</td><td>Basic Auth or restore known-good route</td></tr></tbody></table>

</div></div>---

</div>## Nextcloud Failure

<div id="bkmrk-confirm-vm-address-1" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm VM address `192.168.2.100`.
2. Confirm the 500 GB data disk.
3. Confirm `/mnt/nextcloud-data`.
4. Confirm free disk space.
5. Confirm Docker.
6. Open AIO management by private IP.
7. Confirm PostgreSQL.
8. Confirm Redis.
9. Confirm Nextcloud.
10. Confirm Apache.
11. Confirm TCP 11000.
12. Confirm Traefik.
13. Use the local administrator if OIDC fails.
14. Restore both virtual disks if recovery is required.

</div>Do not start normal Nextcloud operation if the data mount is missing.

<div id="bkmrk--6" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Vikunja Failure

<div id="bkmrk-confirm-vm-address-1-1" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm VM address `192.168.2.121`.
2. Confirm Docker.
3. Confirm PostgreSQL.
4. Confirm the Vikunja container.
5. Confirm TCP 3456.
6. Test `/health`.
7. Run the Vikunja doctor command.
8. Check attachment permissions.
9. Check Traefik.
10. Check Authentik after application health is restored.

---

</div>## BookStack Failure

<div id="bkmrk-confirm-the-bookstac" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm the BookStack VM.
2. Confirm the current private IP.
3. Confirm Docker.
4. Confirm MariaDB.
5. Confirm the BookStack container.
6. Confirm TCP 6875.
7. Review the Laravel log.
8. Confirm `APP_URL`.
9. Confirm the application key.
10. Confirm Traefik.
11. Switch to standard authentication if OIDC blocks access.
12. Restore the VM if application data is damaged.

</div>The application key and database must be restored together.

<div id="bkmrk--7" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Uptime Kuma Failure

<div id="bkmrk-confirm-vm-address-1-2" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm VM address `192.168.2.115`.
2. Confirm Docker.
3. Confirm the Uptime Kuma container.
4. Confirm persistent data.
5. Confirm embedded database health.
6. Confirm container DNS.
7. Test TCP 3001.
8. Confirm Traefik.
9. Confirm Authentik Proxy Provider.
10. Restore local authentication if proxy access fails.

</div>The absence of monitoring does not necessarily mean every monitored system is down.

<div id="bkmrk--8" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Backup HDD Failure

<div id="bkmrk-confirm-%2Fdev%2Fsda.-co" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Confirm `/dev/sda`.
2. Confirm `/dev/sda1`.
3. Confirm filesystem UUID.
4. Confirm `/etc/fstab`.
5. Confirm `/mnt/pve/pve-backup`.
6. Confirm Proxmox storage state.
7. Review SMART health.
8. Stop scheduled backups if the filesystem is unsafe.
9. Replace the backup HDD if necessary.
10. Create a fresh backup set immediately after replacement.

</div>Do not allow backups to write into an unmounted directory on the root filesystem.

<div id="bkmrk--9" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## User-Data NVMe Failure

Expected impact includes Nextcloud user data and any other data disks stored on `user-data`.

Response:

<div id="bkmrk-stop-affected-guests" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Stop affected guests.
2. Avoid further writes.
3. Confirm the physical NVMe.
4. Review NVMe health.
5. Confirm the `lvm1` volume group.
6. Confirm the `data` thin pool.
7. Identify the latest valid backups.
8. Replace the failed storage if required.
9. Recreate the `user-data` storage.
10. Restore affected data disks.
11. Validate guests offline.
12. Return services to production.

---

</div>## Authentication Lockout

### Proxmox

Use `root@pam`.

### Nextcloud

Use the local administrator and direct local-login path.

### BookStack

Set `AUTH_METHOD` to `standard`, recreate the BookStack container, and use the local administrator.

### Uptime Kuma

Restore the direct Traefik route, allow temporary private access, and re-enable local authentication.

### Traefik Dashboard

Restore the previous dashboard YAML or use retained Basic Auth.

### Authentik

Use the local Authentik administrator or restore Authentik from backup.

<div id="bkmrk--10" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Restore Validation

Before replacing production with a restored guest:

<div id="bkmrk-restore-under-a-temp" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Restore under a temporary guest ID.
2. Disconnect the network adapter.
3. Start through the Proxmox console.
4. Confirm operating-system boot.
5. Confirm all intended disks.
6. Confirm mount points.
7. Confirm Docker.
8. Confirm database.
9. Confirm application data.
10. Confirm configuration and secrets.
11. Shut down the test clone.
12. Schedule the production cutover.
13. Shut down the failed production guest.
14. Assign the expected network identity to the restored guest.
15. Start the restored guest.
16. Test direct backend access.
17. Test Traefik.
18. Test Authentik.
19. Test external access.
20. Update documentation.

---

</div>## Emergency Credentials

The password manager must contain:

<div id="bkmrk-proxmox-root%40pam-pas" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">- Proxmox `root@pam` password.
- Authentik administrator credentials.
- Nextcloud local administrator credentials.
- BookStack break-glass administrator credentials.
- Uptime Kuma administrator credentials.
- Traefik Basic Auth credentials.
- VPS SSH private key.
- Home-infrastructure SSH keys.
- WireGuard configuration and recovery keys.
- WHC cPanel credentials.
- SMTP credentials.
- Database recovery credentials where required.

</div>Do not store the actual credential values in BookStack.

BookStack should record only the credential purpose and storage location.

<div id="bkmrk--11" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Recovery Priority

Restore services in this order:

<div id="bkmrk-proxmox-ve.-storage-" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">1. Proxmox VE.
2. Storage pools.
3. Pi-hole.
4. WireGuard Gateway.
5. Authentik database and Authentik.
6. Traefik.
7. Uptime Kuma.
8. Nextcloud.
9. Vikunja.
10. BookStack.
11. Future media and game services.

---

</div>## Incident Record

After a significant incident, record:

<div id="bkmrk-date-and-time.-affec" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">- Date and time.
- Affected services.
- Initial symptoms.
- Root cause.
- Recovery actions.
- Backup used.
- Data loss, if any.
- Configuration changes.
- Monitoring gaps.
- Preventive actions.
- Documentation changes.

</div>Do not include passwords, tokens, private keys, or session cookies.

<div id="bkmrk--12" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">---

</div>## Final Recovery Checklist

<div id="bkmrk-proxmox-management-w" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">- Proxmox management works.
- Required storage is active.
- Backup HDD is mounted.
- Pi-hole resolves internal and public names.
- WireGuard handshake exists.
- VPS ingress works.
- Authentik readiness returns HTTP 200.
- Traefik routes load.
- Certificates are valid.
- Uptime Kuma is healthy.
- Nextcloud data disk is mounted.
- Nextcloud file access works.
- Vikunja health returns HTTP 200.
- BookStack documentation opens.
- OIDC works.
- Local break-glass accounts work.
- Public access works externally.
- New backup completes successfully.
- Documentation is updated.

---

</div>## Document Control

<div id="bkmrk-owner%3A-bryan-gagne-p" style="font-family: 'Segoe UI'; font-size: 14px; font-style: normal; font-weight: 400; line-height: 20px;">- **Owner:** Bryan Gagne-Plante
- **Primary recovery platform:** Proxmox VE
- **Primary backup storage:** `pve-backup`
- **Backup mount:** `/mnt/pve/pve-backup`
- **Offsite backup:** Not configured
- **Last verified:** YYYY-MM-DD
- **Last disaster-recovery exercise:** YYYY-MM-DD
- **Last full restore test:** YYYY-MM-DD
- **Last authentication-recovery test:** YYYY-MM-DD
- **Last backup-HDD review:** YYYY-MM-DD
- **Known limitations:** One Proxmox host, one local backup disk, and no offsite recovery copy

</div>