02 - Core Infrastructure

• Proxmox VE
• Public VPS Ingress
• WireGuard Gateway
• Pi-hole
• Traefik
• Authentik
• Uptime Kuma

Proxmox VE

Proxmox VE

Purpose

Proxmox VE is the primary hypervisor for the Bytek homelab.

Proxmox hosts and manages:

Proxmox is a critical management system and must remain accessible independently of Pi-hole, Traefik, and Authentik.

Service Information

Management interface:

Open Proxmox VE

Access Policy

The Proxmox management interface must remain private.

Approved access paths include:

The following access methods are prohibited:

Pi-hole resolves pve.bytek.ca directly to 192.168.2.254.

DNS Design

The Proxmox host intentionally uses Quad9 at 9.9.9.9 instead of Pi-hole.

This prevents the hypervisor from depending on a guest VM or LXC for its own DNS service.

The DNS behavior differs depending on the requester:

Proxmox may therefore reach Authentik through the public VPS, WireGuard, and Traefik path.

If that path becomes unreliable for OIDC, a single static entry can map portal.bytek.ca to 192.168.2.182 in the Proxmox /etc/hosts file without changing the general DNS resolver.

Browser DNS Requirement

The private hostname pve.bytek.ca exists only in Pi-hole.

Firefox-based browsers such as Zen may bypass Pi-hole when enhanced DNS protection or DNS over HTTPS is forced.

Recommended browser configuration:

If the private hostname does not load but the IP address works, verify browser DNS protection before changing Proxmox.

Useful Windows checks:

Expected results:

Authentication Model

Normal Authentication

Normal administration uses native OpenID Connect with Authentik.

The matching Authentik provider requires:

Required Authentik bindings:

A user must satisfy both the group and policy requirements.

Authentik Scope Mappings

The Proxmox Authentik provider must include the following selected scope mappings:

Missing scope mappings can allow the browser authorization step to succeed while causing Proxmox to fail at the Authentik userinfo endpoint.

A previously observed failure was:

Selecting the expected mappings in the Authentik provider resolved the authentication flow.

Authentication and Authorization Are Separate

Successful Authentik authentication creates the Proxmox user but does not grant access to VMs, containers, nodes, or storage.

A newly created OIDC user may initially see an empty interface.

The user requires an explicit Proxmox access-control entry.

For the primary administrator:

The username normally ends with the realm:

Do not accidentally grant the permission to the PAM version of the username.

Break-Glass Authentication

The local Proxmox PAM account must remain available.

Never:

The root@pam account should be tested periodically in a private browser session.

Web Service Health

The Proxmox web interface is provided by pveproxy.

Check service state with:

systemctl is-active pveproxy

Expected result:

Check the listener with:

ss -lntp | grep ':8006'

Expected result:

Test locally with:

curl -k https://127.0.0.1:8006/

A 501 method HEAD not available response from a command using curl -I does not indicate an outage. That response confirms the HTTPS service was reached, but the endpoint does not support the HTTP HEAD method.

Network Validation

Test from another LAN VM:

nc -zv -w 5 192.168.2.254 8006

A successful result confirms TCP reachability.

Test HTTPS by IP:

curl -k https://192.168.2.254:8006/

Test HTTPS by private hostname:

curl -k https://pve.bytek.ca:8006/

If the hostname fails but the IP works, investigate DNS rather than pveproxy.

OIDC Validation

Verify that Proxmox can retrieve Authentik discovery information:

curl -fsS https://portal.bytek.ca/application/o/proxmox/.well-known/openid-configuration

The response should contain OIDC metadata such as:

A request to the userinfo endpoint without a bearer token should return an authentication error.

That response is expected and proves the endpoint is reachable.

OIDC Troubleshooting

Review recent Proxmox authentication logs with:

journalctl -u pvedaemon --since "10 minutes ago" --no-pager

Look for terms such as:

Authentication Failure 401

Possible causes include:

Failed to Contact Userinfo Endpoint

Check:

Login Works but No VMs Appear

The OIDC user has not been assigned a Proxmox ACL.

Assign the required role at / with propagation enabled.

Private Hostname Does Not Resolve

Check:

Firewall Policy

Proxmox TCP port 8006 should accept traffic only from approved management sources.

Potential management sources include:

Do not create an internet-wide accept rule for TCP 8006.

SSH should use:

Storage Attached to Proxmox

Review storage state using:

pvesm status

All required storage should show as active.

Backup System

Proxmox uses its built-in scheduled backup system.

Backups are written to:

The general job uses:

The backup mount uses mount-point validation to prevent writing backup data to the Proxmox root filesystem if the HDD is unavailable.

Snapshots

Snapshots are used only for short-term rollback.

Appropriate snapshot cases include:

Snapshots are not backups.

Delete snapshots after the associated change has been validated.

Long-lived snapshots consume thin-pool capacity and increase storage risk.

Planned GPU Passthrough

The RTX 2060 is planned for a dedicated Plex VM.

Before passthrough:

A VM with passed-through hardware cannot use normal live migration.

Proxmox Recovery

If Authentik authentication fails:

If the web interface fails:

If the system disk fails:

Validation Checklist

Document Control

Public VPS Ingress

Purpose

The public VPS is the internet-facing entry point for applications hosted on the Bytek home network.

The VPS prevents individual home services from being exposed directly through the Bell router. Public HTTPS traffic reaches the VPS first, then travels through an encrypted WireGuard tunnel to the home network.

The VPS provides:

Service Information

Architecture Role

The VPS does not host the main Bytek applications.

The VPS forwards approved traffic to the home network.

The primary traffic path is:

The VPS must not have direct access to all home-LAN services unless a specific rule requires it.

Public Hostnames

The following public DNS records point to the VPS:

The public hostname pve.bytek.ca must not point to the VPS.

Proxmox remains available only through the LAN or a trusted remote-access method.

Public HTTPS Flow

External HTTPS requests follow this sequence:

The VPS does not need separate public ports for every HTTPS application because Traefik distinguishes applications by hostname.

WireGuard Connection

The VPS maintains a WireGuard peer relationship with the home WireGuard gateway.

The tunnel is used for:

The existing VPS tunnel must remain separate from any future laptop or phone VPN.

Do not reuse:

WireGuard Health

Review the tunnel using:

sudo wg show

Verify:

A recent handshake confirms peer connectivity but does not prove that application forwarding works.

Test the complete application path separately.

VPS Firewall Policy

The VPS firewall should allow only required public and administrative traffic.

Required Public Access

Traffic That Must Remain Private

Do not expose the following home ports through the VPS:

New public ports must be documented before being added.

SSH Administration

VPS management uses SSH key authentication.

Recommended SSH controls include:

Confirm the current SSH listener using:

ss -lntp

Confirm SSH service health using:

systemctl is-active ssh

Expected result:

Do not place the private SSH key in BookStack.

Public DNS Management

Public DNS records are managed through WHC cPanel.

When adding a new public application:

Do not point public DNS directly to:

Public Application Onboarding Checklist

Before exposing a new application through the VPS, confirm:

Service Validation

Confirm the VPS Public Address

Verify that public DNS resolves application hostnames to:

Confirm the WireGuard Tunnel

Use:

sudo wg show

Expected:

Confirm Public Port 443

Use:

ss -lntp

Confirm that the intended ingress service or forwarding mechanism is listening on TCP port 443.

Confirm Routing to the Home Network

Test the intended WireGuard peer or private tunnel address.

A successful tunnel ping confirms network reachability but not application-level routing.

Confirm an Application Through the VPS

From an external connection, open a public application such as:

Open Authentik

The request should reach Authentik without exposing the private backend port.

External Testing

External testing must use a network outside the home LAN.

Suitable test methods include:

Testing from the home LAN may use Pi-hole split DNS and therefore bypass the VPS.

To validate public ingress, confirm the client resolves the hostname to 38.29.213.101.

Monitoring

Recommended Uptime Kuma monitors include:

Monitoring should distinguish:

A successful ping does not prove HTTPS or WireGuard forwarding is functioning.

Logging

Review logs for:

Useful sources may include:

Do not copy active tokens, private keys, or full authentication headers into BookStack.

Failure Impact

If the VPS is unavailable:

The VPS is therefore critical for external access but not for normal LAN access.

Common Failure Scenarios

Public DNS Resolves Incorrectly

Symptoms:

Check:

WireGuard Handshake Is Missing

Symptoms:

Check:

Tunnel Works but Application Fails

Symptoms:

Check:

Internal Access Works but External Access Fails

This usually indicates a problem with:

Internal split DNS bypasses those components.

External Access Works but Internal Access Fails

This usually indicates a problem with:

Recovery Procedure

If all public applications become unavailable:

If SSH is unavailable:

Backup and Recovery Data

Preserve the following VPS information:

Private keys must remain in a password manager or encrypted backup.

BookStack should record the location of secrets, not the secret values.

Security Review Checklist

Document Control

WireGuard Gateway

Purpose

The WireGuard gateway connects the public VPS to the Bytek home network through an encrypted tunnel.

The gateway allows public HTTPS traffic to reach Traefik without exposing individual home services directly through the Bell router.

The WireGuard gateway provides:

Service Information

Architecture Role

The WireGuard gateway sits between the public VPS and the home LAN.

When an external user opens a hosted application:

The WireGuard gateway does not host the applications.

The gateway only provides secure transport and controlled forwarding.

Primary Traffic Flow

The normal public application path includes:

Only required application traffic should pass through this path.

WireGuard Interface

The WireGuard interface contains:

The existing interface name, tunnel subnet, listening port, and peer address should be recorded after checking the active configuration.

Useful checks:

If the interface is not named wg0, replace wg0 with the actual interface name.

Do not copy the private key into BookStack.

Key Management

WireGuard authentication uses public and private key pairs.

Gateway Private Key

The gateway private key:

Gateway Public Key

The gateway public key may be stored in documentation.

The public key identifies the gateway to the VPS peer.

VPS Public Key

The gateway configuration contains the VPS public key.

The VPS private key remains only on the VPS.

Key Rotation

If a private key is exposed:

Allowed IPs

WireGuard Allowed IPs control routing and peer address validation.

The gateway and VPS must use narrow Allowed IP entries.

Allowed IPs should contain only:

Avoid using overly broad routes unless the architecture requires them.

Do not add the full home LAN to a peer merely for convenience without reviewing the security impact.

IP Forwarding

The gateway requires IPv4 forwarding.

Check the current value with:

sysctl net.ipv4.ip_forward

Expected result:

The persistent setting should be stored in a file under /etc/sysctl.d/.

If forwarding is disabled, the WireGuard handshake may succeed while application traffic still fails.

Network Address Translation

The gateway may use NAT when home-LAN devices do not have a route back to the WireGuard tunnel subnet.

NAT allows an internal destination to see the connection as originating from the WireGuard gateway’s LAN address.

Advantages:

Limitations:

NAT rules should be narrow and limited to required source networks, destinations, and ports.

Firewall Policy

The WireGuard gateway must permit only required tunnel traffic.

Required Traffic

Typical required traffic includes:

Restricted Traffic

The VPS should not automatically receive unrestricted access to:

Each additional route should have a documented purpose.

Traefik Forwarding

The primary application destination behind the WireGuard gateway is Traefik.

The gateway should forward approved public HTTPS traffic to Traefik rather than directly to individual application VMs.

This keeps public routing centralized.

Proxmox Management Through the Tunnel

Proxmox management is not part of the normal public ingress path.

If emergency Proxmox access through the VPS is intentionally enabled, use a narrow route such as:

Do not expose Proxmox TCP 8006 directly on the VPS public interface.

The preferred long-term remote-management method is a dedicated road-warrior VPN for the administrator’s laptop or phone.

Separation From a Future Remote-User VPN

The current WireGuard connection is used for VPS ingress.

A future laptop or phone VPN must use:

Do not reuse the VPS key pair or peer configuration for remote-user devices.

The VPS tunnel and a road-warrior VPN serve different purposes.

Service Startup

The WireGuard gateway should start early after a host reboot.

Preferred dependency order:

The WireGuard service should be enabled at boot.

Check the active service using:

systemctl status wg-quick@wg0 --no-pager

If the interface name differs, replace wg0 with the correct name.

Expected service state:

Health Validation

Confirm the Interface

Use:

sudo wg show

Confirm:

Confirm Tunnel Addressing

Use:

ip -br addr

The WireGuard interface should show the expected tunnel address.

Confirm Routes

Use:

ip route

The required tunnel and LAN routes should be present.

Confirm Forwarding

Use:

sysctl net.ipv4.ip_forward

Expected:

Confirm Firewall

If UFW is used:

sudo ufw status numbered

If iptables NAT is used:

sudo iptables -t nat -L -n -v --line-numbers

If nftables is used:

sudo nft list ruleset

Review only the firewall system actually used by the gateway.

End-to-End Validation

A successful WireGuard handshake does not prove application forwarding works.

Validate the complete path.

From the VPS

Test the home WireGuard peer or tunnel gateway address.

Then test Traefik through the tunnel on the intended destination and port.

From an External Client

Use:

Open a public application such as:

Open Authentik

The application should load through:

From the Home LAN

The same hostname should resolve through Pi-hole and bypass the VPS.

This verifies that public and private paths both work.

Monitoring

Recommended Uptime Kuma monitors include:

Monitoring should distinguish between:

Common Failure Scenarios

Handshake Is Missing

Possible causes:

Handshake Exists but Applications Fail

Possible causes:

Internal Applications Work but Public Applications Fail

This usually indicates a problem with:

One Public Application Fails

If other public applications work, the tunnel is probably healthy.

Check:

Tunnel Stops After a Power Failure

Check startup order:

Troubleshooting Order

When public ingress fails:

Do not regenerate WireGuard keys as an early troubleshooting step.

Logging

Useful log sources include:

Review the WireGuard unit using:

journalctl -u wg-quick@wg0 --since "30 minutes ago" --no-pager

Replace the interface name if necessary.

Look for:

Configuration Backup

Protect the following gateway data:

The VM or LXC is included in the Proxmox backup system.

The WireGuard private key should also exist in an encrypted recovery location.

Do not put the private key directly into BookStack.

Recovery Procedure

If the gateway VM or LXC is corrupted:

If the key is unavailable:

Security Checklist

Validation Checklist

Document Control

Pi-hole

Purpose

Pi-hole provides DNS services for the Bytek home network.

Pi-hole is responsible for:

Pi-hole is foundational infrastructure. Pi-hole must remain accessible independently of Traefik and Authentik so DNS can be repaired during an application or identity outage.

Service Information

Architecture Role

Pi-hole provides DNS to home-LAN clients and selected infrastructure containers.

When a LAN device opens a Bytek application:

This allows the same hostname and HTTPS certificate to be used internally and externally.

Pi-hole is not part of the public application path for external users.

External users resolve Bytek applications through WHC public DNS instead.

DNS Responsibilities

Pi-hole provides two types of DNS resolution.

Public Internet Resolution

Pi-hole forwards normal public DNS queries to its configured upstream resolvers.

Examples include:

Local Bytek Resolution

Pi-hole answers selected Bytek application hostnames using private addresses.

These local records create the split-DNS design.

Split-DNS Records

Application hostnames point internally to Traefik.

Proxmox is the exception because Proxmox does not sit behind Traefik.

Public and Private DNS Comparison

The split-DNS model provides:

Proxmox DNS Exception

The Proxmox host intentionally does not use Pi-hole as its general DNS resolver.

Proxmox uses:

This prevents the hypervisor from depending on a guest LXC for DNS.

As a result:

Do not change the Proxmox resolver to Pi-hole without reviewing the dependency impact.

If Proxmox eventually needs a direct local route to Authentik, use a specific /etc/hosts entry for portal.bytek.ca instead of replacing the general resolver.

DHCP Reservations

Infrastructure guests should receive stable addresses through DHCP reservations.

Reservations are based on the virtual NIC MAC address assigned by Proxmox.

Known Reservations

The WireGuard gateway uses 192.168.2.64.

Confirm whether the WireGuard address is configured through a reservation or static guest networking before editing it.

Why Stable Addresses Matter

Infrastructure addresses are referenced by:

An unexpected address change can break several services simultaneously.

Always reserve a service address before treating the address as permanent.

Finding a Guest MAC Address

For a Proxmox VM, inspect the net0 entry using:

qm config <VMID>

For an LXC, inspect the network entry using:

pct config <CTID>

Record the MAC address in Pi-hole when creating the reservation.

Do not manually duplicate a MAC address between guests.

DNS Clients

Pi-hole may provide DNS to:

Not every infrastructure host must use Pi-hole.

Proxmox intentionally uses Quad9.

Other critical systems may also use an independent resolver if avoiding a dependency on Pi-hole is more important than split-DNS access.

Docker DNS

Some application containers require explicit Pi-hole DNS.

This is particularly important for containers that must resolve private Bytek hostnames.

Traefik

The Traefik container explicitly uses:

This was added after Docker’s embedded resolver failed to resolve the Let’s Encrypt ACME endpoint following a power interruption.

Uptime Kuma

The Uptime Kuma container explicitly uses:

This allows Kuma to monitor private Bytek application hostnames.

BookStack

BookStack may use Pi-hole explicitly so the container can resolve Authentik at portal.bytek.ca.

Nextcloud

Nextcloud containers must be able to resolve portal.bytek.ca for Authentik OIDC.

If the host resolves a name but the container does not, inspect the container’s Docker DNS configuration.

Docker DNS Validation

Confirm a container’s configured DNS servers using:

sudo docker inspect --format='DNS={{json .HostConfig.Dns}}' <container-name>

Test hostname resolution from inside a container using:

sudo docker exec <container-name> getent hosts portal.bytek.ca

Expected internal result:

If the command returns no output, container DNS is failing.

Do not change OIDC client IDs or secrets before DNS is proven healthy.

Browser DNS Over HTTPS

Firefox-based browsers, including Zen, can bypass Pi-hole through DNS over HTTPS.

This previously prevented access to:

Pi-hole correctly resolved the private hostname, but Zen attempted to use protected public DNS instead.

Recommended setting:

If a private hostname works by IP but not by name:

Useful Windows commands include:

DNS Validation

Test an Internal Application Record

Use:

nslookup portal.bytek.ca 192.168.2.65

Expected result:

Test Proxmox Resolution

Use:

nslookup pve.bytek.ca 192.168.2.65

Expected result:

Test Public Resolution

Use:

nslookup google.com 192.168.2.65

Expected result:

Test DNS Over TCP

Use:

nc -zv -w 3 192.168.2.65 53

Test DNS Over UDP

Use:

nc -zvu -w 3 192.168.2.65 53

Both TCP and UDP DNS should be available to approved LAN clients.

Service Health

Confirm that the Pi-hole LXC is running from Proxmox.

Confirm the guest address using:

ip -br -4 addr

Confirm DNS listeners using:

ss -lntup | grep ':53'

Confirm the Pi-hole DNS service using the appropriate service command for the installed Pi-hole version.

Review recent service logs if DNS queries fail.

Management Access

Pi-hole management should remain private.

Approved access paths include:

Do not route Pi-hole administration through:

Why Pi-hole Is Not Behind Authentik

Pi-hole is part of the dependency chain for internal Authentik and application resolution.

If Pi-hole management depended on Authentik, the following circular failure could occur:

Pi-hole therefore uses local credentials and private network access.

Upstream DNS

Pi-hole forwards non-local queries to configured upstream resolvers.

The chosen upstream resolver configuration should be documented after verification.

When changing upstream DNS:

Do not assume that an upstream failure means Pi-hole itself is unavailable.

IPv6 Considerations

The Bytek homelab mainly uses IPv4 addressing.

If IPv6 is not intentionally configured end to end:

An incorrect public AAAA record can cause clients to attempt an unavailable path before falling back to IPv4.

Firewall Policy

Pi-hole must accept DNS from approved clients.

Typical requirements include:

Avoid exposing DNS publicly.

A publicly accessible recursive DNS resolver can be abused.

The Bell router must not forward public TCP or UDP port 53 to Pi-hole.

Monitoring

Recommended Uptime Kuma monitors include:

Monitoring should distinguish between:

Common Failure Scenarios

Pi-hole Guest Is Down

Symptoms:

Check:

Public DNS Fails but Local Records Work

Possible causes:

Local Records Fail but Public DNS Works

Possible causes:

VM Host Resolves but Container Does Not

Possible causes:

Recovery may require recreating the container after Pi-hole is working.

One Host Uses the Wrong Address

Possible causes:

Power-Failure Recovery

After an electrical outage, Pi-hole should start before applications that depend on split DNS.

Recommended validation order:

Application containers may need to be recreated or restarted if Docker cached a broken DNS path during startup.

Backup

The Pi-hole LXC is included in the Proxmox scheduled backup job.

The backup should preserve:

A Pi-hole configuration export may be maintained in addition to the Proxmox backup.

The export should be stored securely because it may contain internal network details.

Recovery Procedure

If Pi-hole is unavailable:

If Pi-hole is restored under a temporary CT ID:

Address-Change Procedure

Changing Pi-hole’s address would affect most of the environment.

Before changing 192.168.2.65:

Avoid changing Pi-hole’s address unless required.

Security Checklist

Validation Checklist

Document Control

Traefik

Purpose

Traefik is the central HTTPS reverse proxy for the Bytek homelab.

Traefik receives HTTPS requests, selects the correct application according to the requested hostname, and forwards the request to the appropriate private backend.

Traefik provides:

Service Information

Dashboard:

Open the Traefik Dashboard

Architecture Role

Traefik is the common application-routing layer for both external and internal clients.

External Request Path

Internal Request Path

The same application hostname is used internally and externally.

Application Routing Reference

The BookStack private address must be confirmed and added to this page.

Configuration Locations

The exact certificate and log paths should be confirmed against the current Compose file before relying on them during recovery.

Dynamic Configuration

Each application normally has its own dynamic configuration file.

Known files include:

Separating services into individual files makes configuration changes easier to review and reverse.

Before editing a working file:

Static Configuration

The static Traefik configuration defines platform-level settings such as:

Static configuration changes normally require a Traefik container restart or recreation.

Dynamic routing changes are usually detected automatically by the file provider when file watching is enabled.

Docker DNS

The Traefik container explicitly uses Pi-hole at 192.168.2.65.

This configuration was added after Docker’s embedded resolver failed to resolve the Let’s Encrypt ACME API following a power interruption.

The observed error reported that the Docker resolver at 127.0.0.11 was misbehaving.

Explicit Pi-hole DNS allows Traefik to resolve:

Verify the configured Docker DNS using:

sudo docker inspect --format='DNS={{json .HostConfig.Dns}}' traefik

Expected result:

The container’s internal resolver file may still show Docker’s embedded resolver at 127.0.0.11.

That is normal when Docker forwards requests to the explicitly configured upstream DNS server.

Container Management

Move to the deployment directory before running Compose commands:

cd /opt/traefik

Check the stack:

sudo docker compose ps

View recent logs:

sudo docker compose logs --tail 100 traefik

Follow logs live:

sudo docker compose logs -f traefik

Validate the Compose file:

sudo docker compose config --quiet

Apply normal Compose changes:

sudo docker compose up -d

Recreate Traefik after changing container-level settings such as DNS:

sudo docker compose up -d --force-recreate traefik

Restart Traefik after a static configuration change:

sudo docker compose restart traefik

A normal restart does not apply changes to Docker-level settings such as container DNS, network mode, volume mounts, or environment variables. Those changes require container recreation.

Naming and Routing Standard

Every Traefik application should have:

A route should not contain Authentik ForwardAuth if the application already uses native OIDC and requires direct API or client access.

Applications That Must Not Use ForwardAuth

Do not attach Authentik ForwardAuth to:

Nextcloud

Nextcloud mobile clients, desktop clients, WebDAV, CalDAV, CardDAV, and application authentication must reach Nextcloud directly.

Nextcloud uses native OIDC.

Vikunja

Vikunja uses native Authentik OIDC.

BookStack

BookStack uses native Authentik OIDC.

Authentik

Authentik must never depend on its own ForwardAuth middleware.

Proxmox VE

Proxmox uses native OIDC and remains outside Traefik.

Plex

Plex clients use Plex authentication and Plex-specific APIs.

Applications That Use Authentik Proxy Integration

Traefik Dashboard

The Traefik dashboard uses single-application ForwardAuth.

The dashboard request is validated by the Authentik embedded outpost before access is granted.

Uptime Kuma

Uptime Kuma uses an Authentik Proxy Provider because Uptime Kuma does not provide the same native OIDC integration as the other applications.

Selected status-page and monitoring endpoints may bypass authentication when public access is intentionally required.

Traefik Dashboard Authentication

The dashboard is available at:

Open the Traefik Dashboard

Access is restricted by:

During the initial ForwardAuth validation, Traefik Basic Auth remains enabled as a second authentication layer.

The expected initial flow is:

Basic Auth may be removed only after ForwardAuth and recovery procedures are fully tested.

Authentik ForwardAuth Middleware

The ForwardAuth middleware sends authentication requests to the Authentik embedded outpost.

The outpost backend is hosted by Authentik at 192.168.2.162 on TCP port 9000.

The middleware should trust the forwarded request headers and return Authentik identity headers to the protected application.

Typical identity headers include:

Do not log full authentication headers in a location accessible to untrusted users.

Authentik Outpost Callback Route

The path under /outpost.goauthentik.io/ must remain reachable without the ForwardAuth middleware.

If the callback path is protected by the same middleware, an authentication loop can occur.

The callback router should:

Test the outpost with:

curl -k -I https://proxy.bytek.ca/outpost.goauthentik.io/ping

Expected result:

A normal browser link is not useful for this endpoint because an HTTP 204 response intentionally contains no page content.

Certificate Management

Traefik obtains HTTPS certificates from Let’s Encrypt.

The current design uses:

Each public router should reference the correct certificate resolver.

A router without the certificate resolver may use the Traefik default certificate instead of requesting the intended hostname certificate.

ACME DNS-01 Flow

Certificate issuance follows this sequence:

Traefik must have working DNS before this process can succeed.

Certificate Storage

Certificate state is stored in:

/opt/traefik/data/acme.json

The file must:

Deleting acme.json can remove all stored certificates and account state.

That may trigger unnecessary reissuance and certificate-rate-limit risk.

Certificate Validation

Inspect a certificate served by Traefik using:

openssl s_client -connect 127.0.0.1:443 -servername projects.bytek.ca </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Verify:

Repeat with the relevant hostname when validating another service.

Certificate Troubleshooting

Certificate Missing From the Dashboard

Check:

ACME DNS Failure

A previously observed error reported:

The correction was to configure the Traefik container to use Pi-hole explicitly and recreate the container.

Wrong Certificate Served

Possible causes include:

Application Onboarding

When adding a new public application:

Backend Validation

Before troubleshooting an application through Traefik, test the direct backend from the Traefik VM.

Examples:

Nextcloud

nc -zv -w 5 192.168.2.100 11000

Vikunja

curl -i http://192.168.2.121:3456/health

Authentik

curl -I http://192.168.2.162:9000/-/health/ready/

BookStack

Use the confirmed BookStack IP on TCP port 6875.

A direct backend failure must be resolved before changing the Traefik router.

Local Router Validation

Traefik can be tested locally without relying on public DNS.

Example for Vikunja:

curl -k --resolve projects.bytek.ca:443:127.0.0.1 https://projects.bytek.ca/health

This forces the requested hostname to the local Traefik listener.

A successful response confirms:

Logging

Review the main Traefik log using:

sudo tail -n 100 /opt/traefik/logs/traefik.log

Filter common issues using:

sudo grep -iE 'error|acme|certificate|middleware|service|router|dns' /opt/traefik/logs/traefik.log | tail -n 100

Look for:

If access logging is enabled, use it to distinguish:

Do not copy session cookies, bearer tokens, or authorization headers into BookStack.

YAML Validation

YAML indentation must use spaces, not tabs.

Check dynamic configuration files using:

sudo grep -RnP '\t' /opt/traefik/dynamic/

Expected result:

Review a specific file using:

sudo cat /opt/traefik/dynamic/<filename>.yml

Common YAML problems include:

IP Address Changes

If an application IP changes:

A stale backend address causes gateway errors even when the application itself is healthy.

Firewall Requirements

The Traefik VM must accept:

Application VM firewalls must permit Traefik to reach their backend ports.

Known requirements include:

Do not allow public clients to connect directly to these backend ports.

Monitoring

Recommended Uptime Kuma monitors include:

Monitoring should distinguish between:

Power-Failure Recovery

After a power interruption:

If Traefik started before Pi-hole and DNS remains broken, recreate the Traefik container after Pi-hole is healthy.

Backup

The Traefik VM is included in the Proxmox scheduled backup job.

Protect the following paths:

Secrets must remain in a password manager or encrypted backup.

BookStack should document the secret location, not the secret itself.

Recovery Procedure

If Traefik configuration is broken:

If the VM must be restored:

Security Checklist

Validation Checklist

Document Control

Authentik

Purpose

Authentik provides centralized identity and access management for the Bytek homelab.

Authentik allows users to access multiple applications with one account while applying consistent authentication, multi-factor authentication, group membership, and access policies.

Authentik provides:

Service Information

Authentik portal:

Open Authentik

Architecture Role

Authentik is the trusted identity provider for the Bytek environment.

Applications rely on Authentik to answer these questions:

Authentik does not normally replace application-specific permissions.

For example:

Request Flow

A normal OIDC login follows this process:

Core Components

Authentik Server

The Authentik server provides:

Authentik Worker

The worker performs background tasks such as:

PostgreSQL

PostgreSQL stores Authentik configuration and identity data.

PostgreSQL contains critical information including:

PostgreSQL is the most important Authentik data component to protect.

Redis

Redis supports Authentik’s runtime and task-processing requirements.

Redis is required for normal stack operation but PostgreSQL remains the authoritative persistent data source.

Embedded Outpost

The embedded outpost provides:

Application Portal

The Authentik user portal displays applications available to the signed-in user.

The portal should show only applications that the user is allowed to access.

Normal users may see:

Administrators may additionally see:

Application visibility is controlled by:

Group Model

bytek-users

The bytek-users group grants access to normal user-facing applications.

Applications using this group include:

Users should be added to bytek-users only when access to normal Bytek applications is required.

bytek-admin

The bytek-admin group grants access to administrative applications.

Applications using this group include:

Membership in bytek-admin must be limited to trusted infrastructure administrators.

Administrator Membership

An administrator may belong to both groups:

This allows access to both normal applications and administrative interfaces.

Shared Email-Domain Policy

Applications also use the policy:

The policy validates the user according to the configured Bytek email-domain rules.

The application policy engine is normally set to ALL.

For a normal application, access requires:

For an administrative application, access requires:

Policy Engine Mode

Authentik applications normally use policy engine mode ALL.

With ALL, every enabled binding must pass.

A normal application should therefore have:

An administrative application should have:

Do not attach both bytek-users and bytek-admin to the same application using ALL unless every permitted user must belong to both groups.

An earlier access-denied event occurred because both groups were attached to the same application with ALL.

Application Integration Matrix

Native OIDC Applications

Nextcloud

Nextcloud uses native OIDC so desktop clients, mobile clients, WebDAV, CalDAV, and CardDAV can communicate directly with Nextcloud.

Do not add ForwardAuth to the Nextcloud Traefik router.

Vikunja

Vikunja creates or matches users after a successful Authentik login.

BookStack

BookStack does not automatically link users solely by matching email addresses.

BookStack links an OIDC identity using the token’s sub claim.

The local BookStack administrator must use a different email address from the normal Authentik account unless the account is deliberately linked using the correct external authentication ID.

Proxmox VE

Proxmox authentication and authorization are separate.

Authentik allows the user to authenticate, but Proxmox requires a separate ACL before the user can see or manage VMs.

OIDC Provider Standard

New OIDC providers should normally use:

A signing key is required so applications can validate tokens.

An encryption key should remain empty unless the target application explicitly supports encrypted OIDC tokens.

BookStack login failed when an encryption key was selected.

Clearing the encryption key while retaining the RSA signing key resolved the token-validation failure.

OIDC Scope Mappings

Providers must include the scopes required by the application.

Common scopes include:

The mapping names may appear as Authentik default OAuth scope mappings.

Missing scope mappings can allow the initial browser authorization to complete while causing the application to fail later when requesting profile or userinfo data.

Proxmox Scope Issue

Proxmox initially returned an authentication failure because the Authentik provider did not have the expected scopes selected.

The detailed Proxmox log reported a failure while contacting the userinfo endpoint.

Selecting the required mappings resolved the issue.

Scope Review Procedure

When an OIDC application fails:

Redirect URI Requirements

Redirect URIs must match the application exactly.

A mismatch can cause:

Known Redirect URIs

Do not add unneeded wildcard redirect URIs.

Strict redirect matching is preferred.

Post-Logout Redirects

Where supported, providers should include approved post-logout destinations.

BookStack uses destinations such as:

Post-logout redirects must use the actual production hostname.

Do not use obsolete provisional hostnames such as wiki.bytek.ca.

ForwardAuth Applications

Traefik Dashboard

The Traefik dashboard uses a Proxy Provider configured for single-application ForwardAuth.

The Traefik middleware sends authentication checks to Authentik.

The callback path under /outpost.goauthentik.io/ must remain reachable without applying the same ForwardAuth middleware.

The outpost ping endpoint should return HTTP 204.

Uptime Kuma

Uptime Kuma uses an Authentik Proxy Provider.

Public status-page, badge, push, and asset paths may require selected unauthenticated-path exceptions.

Do not disable Uptime Kuma’s local authentication until the Authentik proxy path has been tested successfully.

Embedded Outpost

The embedded outpost handles Authentik proxy integrations.

Applications assigned to the embedded outpost include:

The outpost must be able to reach each protected application backend.

For Uptime Kuma, Authentik at 192.168.2.162 must be permitted to reach 192.168.2.115 on TCP port 3001.

The outpost callback path must remain accessible through Traefik.

Outpost Health

The outpost health endpoint should return HTTP 204.

For the Traefik dashboard, test:

curl -k -I https://proxy.bytek.ca/outpost.goauthentik.io/ping

Expected result:

A 204 response contains no page body and is expected.

If the endpoint redirects to login or loops repeatedly, review:

Reverse-Proxy Requirements

Traefik must preserve the original request information for Authentik.

Important forwarded values include:

Incorrect forwarded headers can cause:

An Authentik CSRF error previously reported that the CSRF cookie was not set.

When diagnosing a CSRF error, review:

DNS Requirements

Authentik must be reachable from:

Internal Resolution

LAN clients using Pi-hole resolve:

Proxmox Resolution

Proxmox uses Quad9 and normally resolves:

This sends Proxmox’s server-to-server OIDC requests through:

Container Resolution

Application containers that require private split DNS should use Pi-hole at 192.168.2.65.

If an OIDC discovery endpoint becomes unreachable after a power failure, verify DNS from inside the application container before changing provider credentials.

Health Endpoints

Direct Readiness

Direct backend:

http://192.168.2.162:9000/-/health/ready/

Expected result:

Routed Readiness

Public hostname:

Open Authentik Readiness

Expected result:

Embedded Outpost

The outpost ping endpoint should return:

Container Management

Move to the Authentik deployment directory:

cd /opt/authentik

Check container state:

sudo docker compose ps

List Compose service names:

sudo docker compose config --services

View recent logs:

sudo docker compose logs --tail 100

View server logs:

sudo docker compose logs --tail 100 server

View worker logs:

sudo docker compose logs --tail 100 worker

The exact service names should be confirmed using the Compose service listing before running service-specific commands.

Expected Container State

The Authentik stack should have healthy or running equivalents of:

The server should not be considered ready until PostgreSQL and Redis are available.

A running Docker container does not necessarily mean the application is ready.

Use the readiness endpoint for application-level validation.

User Administration

When creating a user:

Do not add normal users to bytek-admin.

Administrator Administration

An Authentik administrator should:

Administrative group membership should be reviewed periodically.

MFA

Authentik provides MFA for applications using OIDC or proxy authentication.

MFA should be required for:

Preserve recovery codes or backup authentication devices.

Do not rely on one physical authenticator without a documented recovery method.

Local Application Recovery Accounts

Authentik must not be the only recovery path.

Proxmox

Use:

Nextcloud

Use:

BookStack

Switch:

Then sign in using the dedicated local administrator.

Uptime Kuma

Restore direct private access and re-enable local authentication.

Traefik

Restore a known-good dashboard file or use retained Basic Auth.

Common Failure Scenarios

Application Shows Permission Denied

Possible causes:

Review the Authentik denial explanation.

A binding shown as None in the explanation may correspond to a direct group-membership check.

Discovery Endpoint Is Unreachable

Possible causes:

Test discovery from the application host and container.

CSRF Cookie Not Set

Possible causes:

Test using a new private browser window.

Token Payload Cannot Be Parsed

Possible cause:

Correction:

User’s Email Is Already Taken

The application may not automatically link an OIDC account to a local account based only on email.

For BookStack:

Authentication Works but Application Is Empty

Authentication succeeded, but application authorization is missing.

For Proxmox:

OIDC Troubleshooting Order

When an OIDC application fails:

Do not recreate the provider until the detailed failure is understood.

Monitoring

Recommended Uptime Kuma monitors include:

Monitoring should distinguish between:

Uptime Kuma

Purpose

Uptime Kuma provides availability monitoring for the Bytek homelab.

Uptime Kuma monitors services at multiple layers so an outage can be isolated quickly.

Monitoring can determine whether a failure is caused by:

Service Information

Dashboard:

Open Uptime Kuma

Architecture Role

Uptime Kuma monitors both private backends and routed application hostnames.

A routed monitor follows this path:

A direct backend monitor bypasses Traefik and connects directly to the application VM.

Using both monitor types helps distinguish infrastructure failures from reverse-proxy failures.

Monitoring Layers

Each important service should have up to three monitoring layers.

VM or LXC Monitor

A ping monitor verifies that the guest is reachable on the network.

A successful ping proves:

A successful ping does not prove that the application is healthy.

Direct Backend Monitor

A direct backend monitor connects to the private application address and port.

A successful backend monitor proves:

A direct backend monitor bypasses:

Routed Application Monitor

A routed monitor uses the normal HTTPS application hostname.

A successful routed monitor proves:

Failure Interpretation

A failed ping monitor does not always mean the VM is down. The guest or Proxmox firewall may block ICMP while allowing application traffic.

Docker Deployment

The Uptime Kuma deployment is managed using Docker Compose.

Deployment directory:

/opt/uptime-kuma

Before running Compose commands, move into the deployment directory using:

cd /opt/uptime-kuma

Check container state using:

sudo docker compose ps

View recent logs using:

sudo docker compose logs --tail 100 uptime-kuma

Follow logs live using:

sudo docker compose logs -f uptime-kuma

Validate the Compose configuration using:

sudo docker compose config --quiet

Apply normal configuration changes using:

sudo docker compose up -d

Recreate the container after changing Docker-level settings such as DNS, volumes, environment variables, or networking using:

sudo docker compose up -d --force-recreate uptime-kuma

Persistent Data

Uptime Kuma stores its application data under /app/data inside the container.

The persistent host mapping should be confirmed from the current Compose file.

The persistent data includes information such as:

The persistent-data directory must remain mounted across container recreation.

Never remove the persistent volume while troubleshooting a container-startup problem.

Database

The current Uptime Kuma deployment uses embedded MariaDB.

Database health is reported by the container health check.

The container should not be considered fully healthy until the database is available.

Check container health using:

sudo docker inspect --format='{{.State.Health.Status}}' uptime-kuma

Expected result:

If the result is starting, allow the health check to continue while reviewing logs.

If the result is unhealthy, inspect the Uptime Kuma logs before restarting or recreating the container.

Docker DNS

The Uptime Kuma container explicitly uses Pi-hole at 192.168.2.65.

This allows Uptime Kuma to resolve internal split-DNS records.

Verify container DNS using:

sudo docker inspect --format='DNS={{json .HostConfig.Dns}}' uptime-kuma

Expected result:

Test an internal hostname from inside the container using:

sudo docker exec uptime-kuma getent hosts portal.bytek.ca

Expected result:

Test another internal hostname using:

sudo docker exec uptime-kuma getent hosts projects.bytek.ca

Expected result:

DNS Failure After a Power Interruption

Docker DNS previously failed after an electrical outage.

Symptoms included:

Recommended recovery sequence:

Do not rebuild Uptime Kuma or delete persistent data because of a DNS-only failure.

Public Dashboard Routing

The Uptime Kuma dashboard is available at:

Open Uptime Kuma

Internal clients resolve status.bytek.ca to Traefik at 192.168.2.182.

External clients resolve status.bytek.ca to the public VPS at 38.29.213.101.

The public path includes:

Authentication Design

Uptime Kuma does not use the same native OIDC design as Nextcloud, Vikunja, BookStack, and Proxmox.

The target design uses an Authentik Proxy Provider.

The expected authentication flow is:

Authentik Application

Only users who satisfy both the group and policy requirements should reach the dashboard.

Embedded Outpost Connectivity

The embedded outpost runs through the Authentik service at 192.168.2.162.

Authentik must be able to reach Uptime Kuma at:

The Uptime Kuma VM firewall must allow:

Test from the Authentik VM using:

curl -I http://192.168.2.115:3001/

A healthy Uptime Kuma response may redirect to /dashboard.

Authentication Migration Safety

Do not disable Uptime Kuma’s local authentication before the Authentik proxy works.

Use this migration order:

During the intermediate state, both Authentik and Uptime Kuma authentication may appear.

That is intentional until the Authentik path is validated.

Disabling Local Authentication

Only disable local Uptime Kuma authentication after Authentik is proven reliable.

Before disabling local authentication:

After disabling local authentication:

Direct Backend Security

When Uptime Kuma local authentication is disabled, direct access to 192.168.2.115:3001 must be restricted.

Otherwise, a LAN client could bypass Authentik and reach an unauthenticated dashboard.

The final firewall should permit TCP port 3001 only from approved sources such as:

Remove temporary direct access after validation.

Public Status Pages

Some Uptime Kuma status pages may be intended for unauthenticated public viewing.

Authentik Proxy Provider exclusions may be required for:

Unauthenticated-path exclusions must be as narrow as practical.

Do not exclude the entire Uptime Kuma API.

After configuring exclusions:

Push Monitors

Push monitors rely on a unique URL that an external process calls to report success.

If Uptime Kuma sits behind Authentik, the push path must remain reachable without an interactive login.

Protect push-monitor URLs as secrets.

Do not publish push URLs in BookStack.

If a push monitor stops updating after enabling Authentik:

Monitor Naming Standard

Use consistent names so the dashboard is easy to scan.

Recommended patterns include:

Examples:

Core Infrastructure Monitors

Recommended core monitors include:

Nextcloud Monitors

Expected routed status:

Vikunja Monitors

Expected health status:

The Vikunja VM firewall must allow Uptime Kuma at 192.168.2.115 to reach TCP port 3456.

BookStack Monitors

Use accepted status codes from 200 through 399 if the application redirects to authentication.

The BookStack VM firewall must allow Uptime Kuma at 192.168.2.115 to reach TCP port 6875.

Authentik Monitors

Expected results:

Traefik Monitors

The dashboard monitor must account for Authentik and Basic Auth redirects if those protections are enabled.

Do not store Basic Auth credentials in BookStack.

Proxmox Monitor Considerations

Proxmox may return a response that is valid but not HTTP 200 for some request methods.

Avoid relying on an unsupported HTTP HEAD request.

A TCP monitor on port 8006 can verify listener availability.

An HTTPS monitor can verify the login page if configured with an appropriate method and accepted status range.

Proxmox remains private and should be monitored using its private address.

Certificate Monitoring

Important public hostnames should have certificate-expiry monitoring.

Recommended hostnames include:

Certificate monitoring should alert before expiration.

A valid application response does not guarantee that certificate renewal automation remains healthy.

Notification Channels

Uptime Kuma can notify administrators when a monitor changes state.

The configured notification methods should be recorded after verification.

Potential channels may include:

Notification credentials must remain outside BookStack.

BookStack should document:

Notification Policy

Recommended notification behavior:

Critical Infrastructure

Immediate notification for:

Applications

Notify after enough failed checks to avoid alerts from brief application restarts.

Maintenance

Create maintenance windows before:

This prevents expected maintenance from generating unnecessary alerts.

Monitor Timing

Monitor intervals should balance timely detection with unnecessary load.

Suggested principles:

These values should be tuned after observing false positives and actual recovery behavior.

Status Pages

Status pages provide a consolidated view of selected monitors.

A public status page should include only services appropriate for public disclosure.

Avoid publishing:

A private administrative status page may include more detail behind Authentik.

Maintenance Windows

Use maintenance windows for planned outages.

Common maintenance events include:

Record:

Firewall Requirements

Uptime Kuma requires outbound access to monitored services.

Application firewalls may require narrow inbound rules from:

Known direct-monitor requirements include:

ICMP must be allowed if ping monitors are used.

A failed ping monitor with a healthy HTTP monitor normally indicates blocked ICMP rather than a service outage.

Health Validation

Check the VM address using:

ip -br -4 addr

Check the container state using:

sudo docker compose ps

Check container health using:

sudo docker inspect --format='{{.State.Health.Status}}' uptime-kuma

Confirm direct access using:

curl -I http://192.168.2.115:3001/

Confirm DNS inside the container using:

sudo docker exec uptime-kuma getent hosts portal.bytek.ca

Confirm the public route using:

curl -I https://status.bytek.ca/

Interpret redirects according to the currently enabled authentication layers.

Logging

Review Uptime Kuma logs using:

sudo docker compose logs --tail 100 uptime-kuma

Follow logs live using:

sudo docker compose logs -f uptime-kuma

Review logs for:

Do not paste access tokens, notification secrets, cookies, or push-monitor URLs into BookStack.

Common Failure Scenarios

Many Hostname Monitors Fail Simultaneously

Likely causes:

Check direct IP monitors before assuming every application failed.

Direct Backends Work but Routed Monitors Fail

Likely causes:

VM Ping Fails but HTTP Works

Likely cause:

The VM is likely available.

Kuma Dashboard Works by IP but Not by Hostname

Check:

Authentik Login Works but Kuma Is Unreachable

Check:

Public Status Page Requires Login

Check:

Push Monitor Stops Working

Check:

Power-Failure Recovery

After a power interruption:

Uptime Kuma should start after the core DNS, identity, and routing services when practical.

Backup

The Uptime Kuma VM is included in the Proxmox scheduled backup job.

Protect:

A stopped Proxmox backup provides the strongest whole-VM consistency for a baseline.

Snapshot-mode backup may be used for routine operation where downtime is undesirable.

Recovery Procedure

If Uptime Kuma is corrupted or unavailable:

If the VM must be restored:

Authentik Recovery

If the Authentik proxy prevents access:

Record the original administrator password securely even after local authentication is disabled.

Security Checklist

Validation Checklist

Document Control