04 - Operations

Backup and Restore
Service Startup Order
Disaster Recovery Checklist

Backup and Restore

Purpose

This page documents the backup and restore process for Bytek VMs, LXCs, application data, and configuration.

The current backup system protects against:

Failed upgrades.
Broken configuration changes.
Guest operating-system corruption.
Accidental deletion.
Application database damage.
Failure of a production NVMe disk, provided the backup HDD remains healthy.

The current backup system does not fully protect against:

Fire.
Theft.
Complete host destruction.
Electrical damage affecting the entire server.
Compromise of both production and backup storage.
Failure of the backup HDD before another copy exists.

Backup Storage

Setting	Value
Proxmox storage ID	`pve-backup`
Mount point	`/mnt/pve/pve-backup`
Physical disk	`/dev/sda`
Partition	`/dev/sda1`
Filesystem	ext4
Approximate capacity	4 TB
Content type	VZDump backup files
Mount protection	Enabled
Current location	Same physical Proxmox server

The backup HDD is dedicated to Proxmox backups.

Do not use the backup HDD for:

Active VM disks.
Nextcloud user data.
Plex media.
Game-server data.
Docker application data.
Temporary downloads.
General file storage.

Mount Validation

Before relying on a backup job, confirm that the backup HDD is mounted.

Use:

findmnt /mnt/pve/pve-backup

The source should be /dev/sda1.

Check capacity using:

df -h /mnt/pve/pve-backup

Check Proxmox storage state using:

pvesm status

The pve-backup storage must show as active.

Mount-point protection prevents Proxmox from writing backup archives into the empty mount directory on the Proxmox root filesystem if the HDD is unavailable.

Scheduled Backup Policy

Setting	Recommended Value
Storage	`pve-backup`
Compression	ZSTD
General backup mode	Snapshot
Initial application baseline	Stop mode where practical
Selection	All production VMs and LXCs
Notification	Configure for failed jobs

Recommended retention:

Retention Rule	Count
Keep last	3
Keep daily	7
Keep weekly	4
Keep monthly	3

Retention must be adjusted if the backup HDD approaches capacity.

Backup Modes

Snapshot Mode

Use snapshot mode for normal scheduled backups when minimizing downtime is important.

Snapshot mode is appropriate for:

Infrastructure services.
Stateless or lightly written services.
Routine nightly backups.
Applications with guest-agent filesystem freeze support.

Stop Mode

Use stop mode for the strongest whole-VM consistency when brief downtime is acceptable.

Stop mode is recommended for:

Initial known-good baselines.
Database-heavy services before major changes.
Backups taken before risky upgrades.
Plex or other VMs with PCI passthrough when snapshot behavior is limited.
Applications where exact application consistency is more important than continuous availability.

Suspend Mode

Suspend mode is not the preferred mode for the current environment.

QEMU Guest Agent

VMs should have the QEMU Guest Agent enabled.

Check a VM configuration using:

qm config <VMID>

The configuration should include the guest agent.

Test guest-agent communication using:

qm guest cmd <VMID> ping

Expected result:

An empty successful response.

Inside the VM, verify:

systemctl is-active qemu-guest-agent

Expected result:

active

The guest agent improves guest coordination during snapshot-mode backups.

Disk Inclusion

A VM disk is included in backup by default unless its configuration contains:

backup=0

Review a VM using:

qm config <VMID>

Confirm every required operating-system and application-data disk is included.

For Nextcloud, both of these must be included:

The 64 GB operating-system disk.
The 500 GB user-data disk.

Large replaceable media disks may eventually be excluded, but only after the exclusion is documented and the application configuration remains protected elsewhere.

Service Backup Coverage

Proxmox Host

The normal guest backup job does not back up the Proxmox host operating system.

Preserve separately:

/etc/pve
Network configuration.
Storage configuration.
Firewall configuration.
Authentik realm settings.
Backup-mount configuration.
VM and LXC inventory.
Recovery credentials.

WireGuard Gateway

Protect:

WireGuard configuration.
Private and public keys.
Peer definitions.
Firewall rules.
NAT rules.
SSH configuration.
Network configuration.

Private keys must remain in encrypted storage, not in BookStack.

Pi-hole

Protect:

Local DNS records.
DHCP reservations.
Pi-hole configuration.
Allow and deny lists.
Upstream DNS settings.
LXC network configuration.

A Pi-hole configuration export may supplement the Proxmox backup.

Traefik

Protect:

/opt/traefik/compose.yml
/opt/traefik/traefik.yml
/opt/traefik/.env
/opt/traefik/dynamic/
/opt/traefik/data/acme.json
Any Basic Auth files.
Certificate API configuration.

Never delete acme.json during normal troubleshooting.

Authentik

Protect:

PostgreSQL data.
Redis configuration.
Compose configuration.
Environment file.
Signing keys.
Applications.
Providers.
Policies.
Groups.
Outpost configuration.
Custom templates and media.

PostgreSQL is the authoritative data source for the Authentik configuration.

Uptime Kuma

Protect:

Persistent /app/data.
Embedded database.
Monitor configuration.
Notification configuration.
Status pages.
Compose file.
Uploaded assets.

Nextcloud AIO

Protect:

64 GB operating-system disk.
500 GB user-data disk.
AIO master-container configuration.
PostgreSQL.
Redis.
Nextcloud configuration.
User files.
Collabora configuration.

The AIO Borg repository is currently unconfigured because no separate backup destination is available.

Vikunja

Protect:

/opt/vikunja/postgres
/opt/vikunja/files
/opt/vikunja/config.yml
/opt/vikunja/.env
/opt/vikunja/compose.yml

BookStack

Protect:

/opt/bookstack/app
/opt/bookstack/database
/opt/bookstack/.env
/opt/bookstack/compose.yml
BookStack application key.
Uploaded images and attachments.
Themes and configuration.

The BookStack application key and database must be restored together.

Application-Level Database Exports

Application-level exports supplement the whole-VM backup.

They do not replace the Proxmox backup.

Vikunja PostgreSQL

Run from /opt/vikunja:

sudo docker compose exec -T db pg_dump -U vikunja vikunja > vikunja-backup.sql

BookStack MariaDB

Run from /opt/bookstack using the database service and credentials defined in the Compose environment.

Prompt for the database password rather than placing the password directly in shell history.

Database exports contain sensitive application data.

Store exports securely and remove obsolete copies.

Manual Baseline Backup

Create a manual backup before:

Major upgrades.
Authentication changes.
Database migrations.
GPU passthrough.
Major firewall changes.
Storage changes.
Disabling a local authentication method.
Removing an old service.

Example pattern:

vzdump <VMID> --storage pve-backup --mode stop --compress zstd

Confirm the task ends with:

TASK OK

List the backup storage using:

pvesm list pve-backup

Restore-Test Procedure

A backup is not fully trusted until it has been restored and inspected.

Select a recent backup.
Restore to a new VM or LXC ID.
Choose an appropriate target storage.
Disconnect the restored guest network adapter.
Start the restored guest using the Proxmox console.
Confirm the operating system starts.
Confirm expected disks are present.
Confirm expected filesystems are mounted.
Confirm Docker starts.
Confirm application containers exist.
Confirm the database is available.
Confirm persistent application data exists.
Shut down the restored clone.
Delete the test clone after documenting the result.

Do not start a restored clone on the live LAN while the production guest is active.

The clone may contain the same:

IP address.
Hostname.
DHCP identity.
Application identity.
Database.
OIDC configuration.
Scheduled tasks.

Nextcloud Restore Test

For Nextcloud, confirm:

The 64 GB OS disk restored.
The 500 GB data disk restored.
/mnt/nextcloud-data is mounted.
/mnt/nextcloud-data/ncdata exists.
Docker is active.
AIO containers exist.
User data is present.

Keep the network adapter disconnected during inspection.

Backup Monitoring

Monitor:

Backup job success.
Backup-HDD mount.
Backup-HDD free space.
SMART health.
Thin-pool usage.
Backup duration.
Unexpected backup-size growth.
Failed guest-agent operations.

Useful checks:

pvesm status
df -h /mnt/pve/pve-backup
findmnt /mnt/pve/pve-backup
smartctl -H /dev/disk/by-id/wwn-0x5000cca097c44d7e

Backup-HDD Health

Stable device identifier:

/dev/disk/by-id/wwn-0x5000cca097c44d7e

Review:

Overall SMART health.
Reallocated sectors.
Pending sectors.
Offline uncorrectable sectors.
CRC errors.
Temperature.
SMART error log.

Pending or uncorrectable sectors require immediate investigation.

Backup Failure Response

If a backup job fails:

Read the Proxmox task log.
Confirm pve-backup is mounted.
Confirm free space.
Confirm the guest exists.
Confirm all required storage is active.
Confirm the QEMU Guest Agent where applicable.
Confirm the VM is not locked.
Confirm no previous backup task is stuck.
Resolve the specific error.
Run a manual test backup.
Confirm TASK OK.
Document the failure and correction.

Do not repeatedly rerun a failing job without reading the task log.

Current Limitations

The current backup architecture has two copies:

Live production data.
Local backup archives on the internal HDD.

Both copies remain:

In the same physical server.
In the same building.
On the same electrical system.
Under the same administrative control.

A future third copy should use:

A trusted remote server.
A dedicated Proxmox Backup Server.
Encrypted Borg.
Encrypted Restic.
Remote PBS synchronization.
Another physically separate destination.

Validation Checklist

pve-backup is mounted.
pve-backup is active in Proxmox.
Backup HDD has sufficient free space.
SMART health is acceptable.
Nightly backup job is enabled.
Retention is configured.
Every production guest is included.
Required virtual disks are included.
Nextcloud OS and data disks are included.
Backup-failure notifications are configured.
A recent manual backup completed with TASK OK.
A restored VM was tested offline.
Recovery credentials are stored securely.
Offsite backup remains documented as outstanding work.

Document Control

Owner: Bryan Gagne-Plante
Backup storage: pve-backup
Mount point: /mnt/pve/pve-backup
Physical backup disk: 4 TB HDD
Compression: ZSTD
Routine mode: Snapshot
Baseline mode: Stop where practical
Offsite backup: Not configured
Last verified: YYYY-MM-DD
Last successful scheduled backup: YYYY-MM-DD
Last manual baseline: YYYY-MM-DD
Last restore test: YYYY-MM-DD
Last SMART review: YYYY-MM-DD
Known limitation: Backup and production storage remain in one physical server

Service Startup Order

Purpose

This page documents the preferred startup and validation order after:

A power interruption.
A planned Proxmox shutdown.
A Proxmox reboot.
Network maintenance.
Storage maintenance.
A full-stack restart.

Starting services in dependency order reduces DNS, OIDC, proxy, and monitoring failures.

Preferred Startup Order

Order	Service
1	Proxmox VE
2	Pi-hole
3	WireGuard Gateway
4	Authentik database and Redis
5	Authentik server and worker
6	Traefik
7	Uptime Kuma
8	Nextcloud AIO
9	Vikunja
10	BookStack
11	Plex, when deployed
12	Game server, when deployed
13	Navidrome, when deployed

1. Proxmox VE

Confirm that the hypervisor completed booting.

Validate:

Management IP is 192.168.2.254.
Network bridge is active.
Storage pools are active.
pveproxy is active.
TCP port 8006 is listening.
Backup HDD is mounted.

Useful checks:

systemctl is-active pveproxy
ss -lntp | grep ':8006'
pvesm status
findmnt /mnt/pve/pve-backup

Do not start troubleshooting application guests until required Proxmox storage is active.

2. Pi-hole

Pi-hole should start before services that require split DNS.

Validate:

Pi-hole owns 192.168.2.65.
TCP port 53 responds.
UDP port 53 responds.
Internal records resolve.
Public records resolve.

Test:

nslookup portal.bytek.ca 192.168.2.65

Expected result:

192.168.2.182

Test:

nslookup google.com 192.168.2.65

Expected result:

A public address.

Containers that started before Pi-hole may need to be restarted after Pi-hole becomes healthy.

3. WireGuard Gateway

Validate:

Gateway owns 192.168.2.64.
WireGuard interface exists.
VPS peer appears.
Recent handshake exists.
Transfer counters increase.
IPv4 forwarding is enabled.
Firewall and NAT rules loaded.

Useful checks:

sudo wg show
sysctl net.ipv4.ip_forward
ip route
sudo ufw status numbered

The WireGuard tunnel must be working before external application access can recover.

4. Authentik Database and Redis

Authentik depends on PostgreSQL and Redis.

Validate the Authentik Compose stack using:

sudo docker compose ps

Confirm:

PostgreSQL is running.
Redis is running.
Persistent data is available.
No storage or permission errors appear.

Do not treat Authentik as ready merely because the server container has started.

5. Authentik Server and Worker

After PostgreSQL and Redis are healthy, validate:

Authentik server.
Authentik worker.
Direct readiness endpoint.
OIDC discovery endpoint.
Embedded outpost.

Direct readiness should return HTTP 200.

The outpost ping should return HTTP 204.

OIDC-dependent applications should not be treated as healthy until Authentik is ready.

6. Traefik

Traefik depends on:

Pi-hole for its configured Docker DNS.
Authentik for ForwardAuth and Proxy Providers.
WireGuard for public ingress.

Validate:

Traefik VM owns 192.168.2.182.
Docker is active.
Traefik container is running.
TCP port 443 is listening.
Pi-hole DNS works inside the container.
Dynamic routers loaded.
Certificates are available.
Authentik route works.

Useful checks:

sudo docker compose ps
sudo docker inspect --format='DNS={{json .HostConfig.Dns}}' traefik
sudo tail -n 100 /opt/traefik/logs/traefik.log

If Traefik started before Pi-hole and DNS remains broken, recreate the Traefik container after Pi-hole is healthy.

7. Uptime Kuma

Start monitoring after DNS, identity, and routing are operational.

Validate:

VM owns 192.168.2.115.
Docker is active.
Container is running.
Container health is healthy.
Embedded database is available.
Container DNS resolves internal hostnames.
Dashboard route works.
Monitors begin recovering.

Do not immediately modify every failed monitor after a reboot. Allow dependencies to recover first.

8. Nextcloud AIO

Before starting the Nextcloud application stack, validate:

VM owns 192.168.2.100.
500 GB data disk is attached.
/mnt/nextcloud-data is mounted.
/mnt/nextcloud-data/ncdata exists.
Docker is active.
AIO management is reachable.

Then confirm:

PostgreSQL.
Redis.
Nextcloud.
Apache.
Notify Push.
Collabora.

Test:

TCP port 11000.
cloud.bytek.ca.
Authentik login.
File access.

Do not run Nextcloud normally if the data disk is not mounted.

9. Vikunja

Validate:

VM owns 192.168.2.121.
Docker is active.
PostgreSQL is healthy.
Vikunja container is running.
/health returns HTTP 200.
Authentik login works.

If the container cannot reach Authentik, confirm DNS before changing OIDC credentials.

10. BookStack

Validate:

VM owns its reserved private address.
Docker is active.
MariaDB is running.
BookStack is running.
Application key is available.
Direct backend responds.
docs.bytek.ca works.
Authentik login works.
Existing pages open.

If OIDC fails, temporarily restore standard authentication rather than rebuilding BookStack.

Future Plex Startup

When Plex is deployed, validate:

Plex VM boots.
RTX 2060 is present.
NVIDIA driver loads.
nvidia-smi works.
Docker has GPU access.
Plex configuration storage is mounted.
Media storage is mounted.
Plex web interface responds.
Hardware transcoding is available.

Do not start Plex if expected media storage is missing.

Future Game-Server Startup

When a game server is deployed, validate:

VM owns its reserved address.
Game-data disk is mounted.
SteamCMD or LinuxGSM files exist.
Game server starts.
Required game ports listen.
World or save data loads.
Public game-port forwarding works.
Monitoring works.

Do not expose RCON or management ports publicly without additional protection.

Docker DNS Recovery

A recurring risk after a full power loss is that containers start before Pi-hole is ready.

Symptoms include:

OIDC discovery endpoint unreachable.
Let’s Encrypt DNS lookup failures.
Internal hostnames failing inside containers.
Uptime Kuma monitors remaining down.
Host DNS works while container DNS fails.

Recovery process:

Confirm Pi-hole.
Confirm DNS from the VM host.
Confirm DNS from the affected container.
Check the container’s explicit DNS configuration.
Restart the affected container.
Recreate the container if Docker-level DNS configuration changed.
Retry the failed operation.

Do not regenerate application secrets because of a DNS-only failure.

Full Validation Checklist

Proxmox web interface works.
Required Proxmox storage is active.
Backup HDD is mounted.
Pi-hole resolves internal records.
Pi-hole resolves public records.
WireGuard handshake exists.
Public VPS ingress works.
Authentik direct readiness returns HTTP 200.
Authentik routed readiness returns HTTP 200.
Authentik outpost returns HTTP 204.
Traefik routers loaded.
Traefik certificates are available.
Uptime Kuma is healthy.
Nextcloud status is healthy.
Nextcloud data disk is mounted.
Collabora works.
Vikunja health returns HTTP 200.
BookStack opens existing documentation.
OIDC login works on all configured applications.
External access works from cellular or another remote network.

Document Control

Owner: Bryan Gagne-Plante
Startup authority: Proxmox VE
DNS dependency: Pi-hole
Public-ingress dependency: VPS and WireGuard
Authentication dependency: Authentik
HTTPS dependency: Traefik
Last verified: YYYY-MM-DD
Last power-failure recovery: YYYY-MM-DD
Last full-stack restart: YYYY-MM-DD
Known issue: Containers may require restart or recreation if started before Pi-hole

Disaster Recovery Checklist

Purpose

This checklist provides a concise response plan for major Bytek homelab failures.

The objective is to restore core infrastructure before application services.

Avoid making several unrelated configuration changes simultaneously.

Initial Assessment

Before changing configuration:

Determine whether the problem affects one service or the entire environment.
Check electrical power.
Check the Proxmox host.
Check the home network.
Check internet connectivity.
Check Pi-hole.
Check WireGuard.
Check Traefik.
Check Authentik.
Check application VMs.
Review Uptime Kuma.
Record the observed symptoms.

Do not rely only on browser errors.

Test direct IPs and backend ports where possible.

Complete Proxmox Host Failure

Confirm server power.
Check the physical console.
Confirm the firmware detects all storage devices.
Confirm the Proxmox system NVMe.
Confirm the user-data NVMe.
Confirm the backup HDD.
Attempt normal Proxmox boot.
Review boot messages.
Confirm network bridge configuration.
Confirm pveproxy.
Confirm TCP 8006.
Use root@pam.

If the system NVMe failed:

Replace the failed system storage.
Install Proxmox.
Restore network configuration.
Restore storage configuration.
Reconnect user-data.
Remount pve-backup.
Restore VMs and LXCs.
Verify local authentication.
Restore the Authentik Proxmox realm.
Test OIDC only after local access works.

Pi-hole Failure

Open Proxmox using 192.168.2.254.
Confirm the Pi-hole LXC is running.
Open the LXC console.
Confirm address 192.168.2.65.
Confirm TCP port 53.
Confirm UDP port 53.
Confirm local DNS records.
Confirm public upstream resolution.
Review Pi-hole logs.
Restore from backup if necessary.

During the outage, access services using private IP addresses where supported.

Do not make Pi-hole recovery depend on Authentik.

WireGuard Failure

Confirm the WireGuard gateway guest.
Confirm address 192.168.2.64.
Confirm the WireGuard interface.
Confirm the VPS peer.
Confirm the latest handshake.
Confirm transfer counters.
Confirm IPv4 forwarding.
Confirm firewall rules.
Confirm NAT rules.
Test Traefik connectivity from the tunnel.
Test a public hostname externally.

If the keys are lost or compromised, rotate the affected peer key pair and update the opposite peer.

Public VPS Failure

Open the provider console.
Confirm the VPS is running.
Confirm public IP 38.29.213.101.
Confirm SSH service.
Confirm firewall rules.
Confirm WireGuard.
Confirm TCP 443.
Confirm forwarding to the home tunnel.
Confirm disk space.
Review system logs.

LAN access should continue through Pi-hole and Traefik while the VPS is unavailable.

Traefik Failure

Confirm the Traefik VM owns 192.168.2.182.
Confirm Docker.
Confirm the Traefik container.
Confirm TCP 443.
Confirm container DNS.
Review Traefik logs.
Check recently changed dynamic YAML.
Search for tabs.
Test the direct application backend.
Restore the previous dynamic file if necessary.
Confirm certificate state.
Do not delete acme.json.

If all applications fail simultaneously but direct backends work, prioritize Traefik, Pi-hole, and certificates.

Authentik Failure

Sign into Proxmox using root@pam.
Confirm the Authentik VM owns 192.168.2.162.
Confirm Docker.
Confirm PostgreSQL.
Confirm Redis.
Confirm Authentik server.
Confirm Authentik worker.
Test direct readiness.
Test routed readiness.
Review Authentik logs.
Review Traefik.
Restore Authentik from backup if necessary.

Use application break-glass access during recovery:

Application	Recovery Method
Proxmox	`root@pam`
Nextcloud	Local administrator
BookStack	Switch to standard authentication
Uptime Kuma	Restore local authentication
Traefik	Basic Auth or restore known-good route

Nextcloud Failure

Confirm VM address 192.168.2.100.
Confirm the 500 GB data disk.
Confirm /mnt/nextcloud-data.
Confirm free disk space.
Confirm Docker.
Open AIO management by private IP.
Confirm PostgreSQL.
Confirm Redis.
Confirm Nextcloud.
Confirm Apache.
Confirm TCP 11000.
Confirm Traefik.
Use the local administrator if OIDC fails.
Restore both virtual disks if recovery is required.

Do not start normal Nextcloud operation if the data mount is missing.

Vikunja Failure

Confirm VM address 192.168.2.121.
Confirm Docker.
Confirm PostgreSQL.
Confirm the Vikunja container.
Confirm TCP 3456.
Test /health.
Run the Vikunja doctor command.
Check attachment permissions.
Check Traefik.
Check Authentik after application health is restored.

BookStack Failure

Confirm the BookStack VM.
Confirm the current private IP.
Confirm Docker.
Confirm MariaDB.
Confirm the BookStack container.
Confirm TCP 6875.
Review the Laravel log.
Confirm APP_URL.
Confirm the application key.
Confirm Traefik.
Switch to standard authentication if OIDC blocks access.
Restore the VM if application data is damaged.

The application key and database must be restored together.

Uptime Kuma Failure

Confirm VM address 192.168.2.115.
Confirm Docker.
Confirm the Uptime Kuma container.
Confirm persistent data.
Confirm embedded database health.
Confirm container DNS.
Test TCP 3001.
Confirm Traefik.
Confirm Authentik Proxy Provider.
Restore local authentication if proxy access fails.

The absence of monitoring does not necessarily mean every monitored system is down.

Backup HDD Failure

Confirm /dev/sda.
Confirm /dev/sda1.
Confirm filesystem UUID.
Confirm /etc/fstab.
Confirm /mnt/pve/pve-backup.
Confirm Proxmox storage state.
Review SMART health.
Stop scheduled backups if the filesystem is unsafe.
Replace the backup HDD if necessary.
Create a fresh backup set immediately after replacement.

Do not allow backups to write into an unmounted directory on the root filesystem.

User-Data NVMe Failure

Expected impact includes Nextcloud user data and any other data disks stored on user-data.

Response:

Stop affected guests.
Avoid further writes.
Confirm the physical NVMe.
Review NVMe health.
Confirm the lvm1 volume group.
Confirm the data thin pool.
Identify the latest valid backups.
Replace the failed storage if required.
Recreate the user-data storage.
Restore affected data disks.
Validate guests offline.
Return services to production.

Authentication Lockout

Proxmox

Use root@pam.

Nextcloud

Use the local administrator and direct local-login path.

BookStack

Set AUTH_METHOD to standard, recreate the BookStack container, and use the local administrator.

Uptime Kuma

Restore the direct Traefik route, allow temporary private access, and re-enable local authentication.

Traefik Dashboard

Restore the previous dashboard YAML or use retained Basic Auth.

Authentik

Use the local Authentik administrator or restore Authentik from backup.

Restore Validation

Before replacing production with a restored guest:

Restore under a temporary guest ID.
Disconnect the network adapter.
Start through the Proxmox console.
Confirm operating-system boot.
Confirm all intended disks.
Confirm mount points.
Confirm Docker.
Confirm database.
Confirm application data.
Confirm configuration and secrets.
Shut down the test clone.
Schedule the production cutover.
Shut down the failed production guest.
Assign the expected network identity to the restored guest.
Start the restored guest.
Test direct backend access.
Test Traefik.
Test Authentik.
Test external access.
Update documentation.

Emergency Credentials

The password manager must contain:

Proxmox root@pam password.
Authentik administrator credentials.
Nextcloud local administrator credentials.
BookStack break-glass administrator credentials.
Uptime Kuma administrator credentials.
Traefik Basic Auth credentials.
VPS SSH private key.
Home-infrastructure SSH keys.
WireGuard configuration and recovery keys.
WHC cPanel credentials.
SMTP credentials.
Database recovery credentials where required.

Do not store the actual credential values in BookStack.

BookStack should record only the credential purpose and storage location.

Recovery Priority

Restore services in this order:

Proxmox VE.
Storage pools.
Pi-hole.
WireGuard Gateway.
Authentik database and Authentik.
Traefik.
Uptime Kuma.
Nextcloud.
Vikunja.
BookStack.
Future media and game services.

Incident Record

After a significant incident, record:

Date and time.
Affected services.
Initial symptoms.
Root cause.
Recovery actions.
Backup used.
Data loss, if any.
Configuration changes.
Monitoring gaps.
Preventive actions.
Documentation changes.

Do not include passwords, tokens, private keys, or session cookies.

Final Recovery Checklist

Proxmox management works.
Required storage is active.
Backup HDD is mounted.
Pi-hole resolves internal and public names.
WireGuard handshake exists.
VPS ingress works.
Authentik readiness returns HTTP 200.
Traefik routes load.
Certificates are valid.
Uptime Kuma is healthy.
Nextcloud data disk is mounted.
Nextcloud file access works.
Vikunja health returns HTTP 200.
BookStack documentation opens.
OIDC works.
Local break-glass accounts work.
Public access works externally.
New backup completes successfully.
Documentation is updated.

Document Control

Owner: Bryan Gagne-Plante
Primary recovery platform: Proxmox VE
Primary backup storage: pve-backup
Backup mount: /mnt/pve/pve-backup
Offsite backup: Not configured
Last verified: YYYY-MM-DD
Last disaster-recovery exercise: YYYY-MM-DD
Last full restore test: YYYY-MM-DD
Last authentication-recovery test: YYYY-MM-DD
Last backup-HDD review: YYYY-MM-DD
Known limitations: One Proxmox host, one local backup disk, and no offsite recovery copy