# Storage Architecture ## Purpose This page documents the physical disks, Proxmox storage pools, logical-volume layout, filesystem mounts, and intended storage roles within the Bytek homelab. The storage architecture separates:
- Proxmox and application operating-system disks. - Large user and application data. - Local Proxmox backups. - Temporary snapshots and rollback points.
The main design goal is to prevent application data, operating-system disks, and backups from competing for the same physical storage role.
---
## Storage Overview
---
## Physical Disk Inventory
Linux device names such as `/dev/sda` can change between boots or hardware changes. Permanent references should use:
- Filesystem UUIDs for mounted filesystems. - `/dev/disk/by-id/` paths for physical-device operations. - Proxmox storage IDs for normal VM and backup management. ---
## System NVMe ### Physical Device
### Partition Layout
### Logical Volumes
### Proxmox Storage The `pve-data` thin pool appears in Proxmox as:
- **Storage ID:** `local-lvm` - **Storage type:** LVM-thin
### Intended Use The system NVMe stores:
- Proxmox VE. - Infrastructure VM and LXC operating-system disks. - Application operating-system disks. - EFI disks. - Small configuration volumes. - Temporary Proxmox snapshots. - Docker application configuration stored inside service VMs.
### Current Workloads The system NVMe currently stores operating-system disks for services such as:
- WireGuard. - Pi-hole. - Uptime Kuma. - Traefik. - Authentik. - Nextcloud. - Vikunja. - BookStack.
### Storage Policy Use `local-lvm` for:
- VM and LXC operating-system disks. - Infrastructure services. - Small persistent application datasets. - VM configuration disks. - Temporary snapshots.
Avoid using `local-lvm` for:
- Large Nextcloud user libraries. - Large Plex media libraries. - Large game-server libraries. - Backup archives. - Long-term media collections. ---
## User-Data NVMe ### Physical Device
### LVM-Thin Layout The user-data NVMe contains:
- `lvm1-data_tmeta` - `lvm1-data_tdata` - `lvm1-data`
These components represent:
- Thin-pool metadata. - Thin-pool data. - The visible LVM-thin pool.
### Proxmox Storage The thin pool appears in Proxmox as:
- **Storage ID:** `user-data` - **Storage type:** LVM-thin - **Volume group:** `lvm1` - **Thin pool:** `data` - **Configured chunk size:** 256 KiB
### Intended Use The user-data NVMe stores:
- Nextcloud user files. - Large application-data disks. - Future game-server data. - Future media-storage testing. - Other user-generated content.
### Current Major Workload Nextcloud has a separate virtual data disk on `user-data`.
The Nextcloud operating system remains on `local-lvm`, while user files remain on `user-data`. This separation makes it easier to:
- Expand the user-data disk. - Monitor data growth. - Keep the operating-system disk smaller. - Apply different backup decisions to application and data disks. - Migrate data disks independently in the future. ---
## Thin Provisioning The `local-lvm` and `user-data` storage pools use LVM-thin provisioning. A thin-provisioned virtual disk does not immediately consume its full nominal capacity. For example, a 500 GB virtual disk initially consumes only the physical blocks that have actually been written. ### Important Consequence The sum of all nominal virtual-disk sizes may exceed the physical capacity of the thin pool. This is safe only when actual usage is monitored. If the thin pool reaches full physical allocation, affected VMs may experience:
- Write failures. - Filesystem corruption. - Application outages. - Database failures. - Guest crashes. ---
## Thin-Pool Monitoring Review Proxmox storage usage using: `pvesm status` Review LVM-thin usage using: `lvs -a -o vg_name,lv_name,segtype,lv_attr,lv_size,data_percent,metadata_percent` Monitor both:
- **Data percent** - **Metadata percent**
Suggested operational thresholds:
These thresholds are operational recommendations and should be adjusted based on growth rate and workload importance.
---
## Backup HDD ### Physical Device
### Permanent Mount
The filesystem UUID is used instead of `/dev/sda1` because Linux device letters can change. ### Mount Options The backup filesystem uses:
- Read-write mounting. - `noatime` to reduce unnecessary metadata writes.
### Mount-Point Protection The Proxmox storage uses mount-point validation. This prevents Proxmox from treating the empty directory `/mnt/pve/pve-backup` as valid storage when the HDD is not mounted. Without this protection, a failed HDD mount could cause backups to be written to the Proxmox root filesystem. That could fill the root filesystem and disrupt the hypervisor.
---
## Backup-HDD Intended Use The backup HDD stores:
- Scheduled VM backups. - Scheduled LXC backups. - Manual baseline backups. - Upgrade rollback backups. - Restore-test source archives.
The backup HDD must remain dedicated to backups.
---
## Prohibited Backup-HDD Uses Do not use the backup HDD for:
- Active VM disks. - Active LXC root filesystems. - Nextcloud user files. - Plex media. - Steam game-server data. - Docker application data. - Download staging. - General file storage. - Temporary application caches.
Mixing active workloads with backups would:
- Increase disk contention. - Increase wear. - Reduce available retention. - Expand the failure impact. - Make recovery planning less predictable. ---
## Storage Placement Rules
---
## Future Plex Storage The RTX 2060 Plex VM should not use the backup HDD for media. Recommended placement:
- Plex operating system on `local-lvm`. - Plex configuration and metadata on NVMe-backed storage. - Plex transcode directory on temporary fast storage. - Movies and television content on future dedicated media storage. - Music content on future shared media storage if Navidrome is deployed.
A temporary media disk may be created on `user-data` for initial testing. Long-term media storage should ideally use:
- A dedicated HDD. - A mirrored HDD pair. - A dedicated storage pool. - A NAS or storage server.
Media capacity planning must account for continued Nextcloud growth on `user-data`.
---
## Future Game-Server Storage A dedicated game-server VM should use:
- Operating-system disk on `local-lvm`. - Game data or world-save disk on `user-data`. - Backup enabled for unique save data. - Application-level save backups where supported.
Steam-installed server binaries can usually be downloaded again. Unique game-world data, configuration files, allowlists, bans, and mods require backup.
---
## Storage Expansion Principles Before adding physical storage, decide whether the new disk is intended for:
- Operating-system workloads. - Application data. - Media. - Backups. - Redundancy. - Offsite synchronization.
Avoid adding unrelated physical disks into an existing unmirrored LVM volume group merely to increase capacity. Safer future expansion patterns include:
1. Create a new independent Proxmox storage pool. 2. Move selected VM disks to the new pool. 3. Replace a smaller disk with a larger disk. 4. Add mirrored storage for critical data. 5. Add a dedicated media-storage pool. 6. Add a dedicated Proxmox Backup Server. 7. Add an encrypted offsite backup target. ---
## Failure-Domain Considerations ### System NVMe Failure Expected impact:
- Proxmox root filesystem may be lost. - Infrastructure VM and LXC operating-system disks may be lost. - The user-data NVMe and backup HDD may remain physically intact.
Recovery would require:
- Proxmox reinstallation or system-disk restoration. - Reconnection of existing storage. - VM and LXC restoration from `pve-backup`.
### User-Data NVMe Failure Expected impact:
- Nextcloud user-data disk may be lost. - Other large application-data disks may be lost. - VM operating-system disks on `local-lvm` may continue working. - Backups on the HDD may remain available.
### Backup HDD Failure Expected impact:
- Live services continue running. - New backup jobs fail. - Existing local recovery points are lost. - There is no current offsite replacement copy.
### Complete Host Failure Expected impact:
- All running services become unavailable. - The internal backup HDD may be inaccessible until moved or the host is repaired. - A severe electrical or physical event could affect all three storage devices.
This is the primary reason a future offsite or separate-host backup is required.
---
## Snapshot Policy Proxmox snapshots are used as short-term rollback points. Recommended snapshot use cases:
- Before a major operating-system upgrade. - Before changing Authentik integration. - Before changing Traefik routing or middleware. - Before changing a Docker stack. - Before database migrations. - Before a major application upgrade. - Before enabling GPU passthrough.
Snapshots are not backups because:
- Snapshots remain on the same storage. - A physical disk failure destroys the snapshot with the live disk. - Long-lived snapshots consume thin-pool capacity. - Snapshot growth can create unexpected storage pressure.
Delete unnecessary snapshots after the associated change has been validated. Avoid snapshots with RAM unless restoring the exact running state is required.
---
## Storage Validation ### Confirm Proxmox Storage Use: `pvesm status` Expected storage IDs include:
- `local` - `local-lvm` - `user-data` - `pve-backup`
### Confirm Backup Mount Use: `findmnt /mnt/pve/pve-backup` The source should be `/dev/sda1`. ### Confirm Backup Capacity Use: `df -h /mnt/pve/pve-backup` ### Confirm Physical Devices Use: `lsblk -e7 -o NAME,PATH,SIZE,TYPE,FSTYPE,PTTYPE,MOUNTPOINTS,MODEL,SERIAL,WWN` ### Confirm Thin-Pool Utilization Use: `lvs -a -o vg_name,lv_name,segtype,lv_attr,lv_size,data_percent,metadata_percent` ### Confirm Storage Configuration Use: `cat /etc/pve/storage.cfg` Do not store credentials or secrets in the documentation when recording command output.
---
## Disk-Health Monitoring The backup HDD should be monitored using SMART. Stable backup-HDD identifier: `/dev/disk/by-id/wwn-0x5000cca097c44d7e` Useful checks:
- `smartctl -H /dev/disk/by-id/wwn-0x5000cca097c44d7e` - `smartctl -A /dev/disk/by-id/wwn-0x5000cca097c44d7e` - `smartctl -x /dev/disk/by-id/wwn-0x5000cca097c44d7e`
Important attributes include:
- Reallocated sector count. - Current pending sector count. - Offline uncorrectable sector count. - UDMA CRC error count. - Temperature. - Power-on hours. - SMART error log.
Pending or uncorrectable sectors require immediate review. NVMe health should also be reviewed periodically using the appropriate NVMe SMART tools.
---
## Capacity Review Checklist Perform a storage review when:
- Nextcloud user data grows significantly. - Plex media is introduced. - A game-server data disk is created. - Backup retention consumes unexpected space. - Thin-pool utilization exceeds 70 percent. - Snapshot count increases. - A disk reports SMART warnings. - Another physical disk is being considered.
Review:
1. Physical free capacity. 2. Thin-pool data utilization. 3. Thin-pool metadata utilization. 4. Snapshot count. 5. Backup-HDD free space. 6. Nextcloud data growth. 7. Media-storage requirements. 8. Restore-capacity requirements. 9. Backup-retention effectiveness. ---
## Current Limitations The current storage architecture does not provide disk redundancy. The system NVMe, user-data NVMe, and backup HDD are independent single devices. The architecture protects against some logical failures through backups, but does not provide uninterrupted operation after a physical disk failure. The local backup HDD is in the same:
- Physical server. - Building. - Power environment. - Administrative security boundary.
Future priorities include:
- Dedicated media storage. - A third backup copy. - Offsite encrypted backups. - Separate Proxmox Backup Server hardware. - Storage redundancy for critical user data. ---
## Document Control
- **Owner:** Bryan Gagne-Plante - **Last verified:** YYYY-MM-DD - **System storage:** 1 TB NVMe on `local-lvm` - **Application-data storage:** 4 TB NVMe on `user-data` - **Backup storage:** 4 TB HDD on `pve-backup` - **Storage redundancy:** None - **Offsite backup:** Not configured - **Recovery tested:** Partial - **Known limitations:** Single-device storage tiers and one physical host