Hardening the Homelab: Enterprise UniFi Topology, Encrypted DNS & Policy Routing

Modern operating systems, mobile devices, and browsers are increasingly hostile to local network observability. Operating systems and runtimes now routinely ship with opportunistic or forced DNS-over-HTTPS (DoH) bootstraps. While beneficial over hostile public coffee-shop Wi-Fi, within a segregated engineering lab this behavior breaks local split-horizon DNS resolution, bypasses IDS/IPS boundary policies, and introduces uncontrollable telemetry egress paths.

To re-assert control without compromising upstream privacy, I re-architected my edge network topology around a UniFi Gateway Max (UXG-Max) paired with a multi-instance, containerized AdGuard Home resolver. The resulting design enforces transparent L4/L7 interception across all VLANs, closes encrypted DNS leakage vectors, and maintains mutual TLS to upstream authoritative recursive resolvers over DNS-over-QUIC (DoQ) and DNS-over-HTTPS (DoH).

1. Network Topology & Zone Matrix

The infrastructure divides workloads into discrete broadcast domains using 802.1Q tags managed by the UniFi Gateway Max, terminated on an enterprise 2.5GbE switching core. Inter-VLAN traffic follows an explicit zero-trust model: default drop across all security zones, with stateful return exceptions via conntrack.

                               [ WAN Gateway / Fiber ONT ]
                                            │
                                            ▼
                       [ UniFi Gateway Max (UXG-Max) ]
                       [ Routing, NAT, L4 Firewall   ]
                                     │
      ┌──────────────────────────────┼──────────────────────────────┐
      │                              │                              │
      ▼                              ▼                              ▼
[ VLAN 10: TRUSTED ]           [ VLAN 20: INFRA ]             [ VLAN 30: IOT ]
192.168.10.0/24                192.168.20.0/24                192.168.30.0/24
- Workstations                 - AdGuard Home (Primary: .5)   - Smart Home Hubs
- Laptops                      - AdGuard Home (Replica: .6)   - Streaming Devices
- Mobile (Authorized)          - Unraid / TrueNAS             - IP Cameras (Isolated)
                               - Docker / Kube Node
                                     │
      ┌──────────────────────────────┴──────────────────────────────┐
      │                                                             │
      ▼                                                             ▼
[ VLAN 40: GUEST ]                                            [ VLAN 50: LAB ]
192.168.40.0/24                                               192.168.50.0/24
- Ephemeral Clients                                           - Sandboxed VMs
- Complete Isolation                                          - CI/CD Runners

Zone Security Policy Matrix

2. The Interception Engine: Neutralizing DNS Bypass Vectors

Hardening local DNS requires addressing two persistent evasion patterns: Legacy Plaintext Port 53 Evasion and Opportunistic Port 853/443 Evasion.

Vector A: Hardcoded Port 53 Resolvers

Chromecast, smart TVs, and IoT appliances frequently ignore DHCP Option 6 (domain-name-servers) and hardcode static queries to 8.8.8.8 or 1.1.1.1.

On the UniFi Gateway Max, we enforce a DNAT redirect that transparently rewires any outbound UDP/TCP 53 request to the AdGuard Home instance at 192.168.20.5:

# UniFi Controller Firewall Rule Formulation:
# Rule Name: REDIRECT_PORT_53_TO_ADGUARD
# Action: Port Forward / DNAT
# Source: Any Local Subnet (!VLAN 20)
# Port: 53 (TCP/UDP)
# Destination Target: 192.168.20.5:53

To eliminate loops, the DNS server host (192.168.20.5) is excluded from this NAT rule.

Vector B: Hardcoded Encrypted DNS (DoT / DoQ / DoH)

Modern browsers (Chrome Secure DNS, Safari Private Relay) and Apple/Android devices attempt to upgrade to encrypted transports over ports 853 (DNS-over-TLS / DNS-over-QUIC) or 443 (DoH). If permitted, traffic passes through the gateway as encrypted TLS/QUIC payloads directly to commercial CDNs, blinding local audit logs.

We implement a three-tier block to dismantle this evasion:

[ Client Device ]
        │
        ├─► [ Port 853: DoT / DoQ Request ] ──► [ UXG-Max Firewall Drop ]
        │
        ├─► [ Port 443: Known DoH Providers ] ──► [ IP/CIDR Blackhole List Drop ]
        │
        └─► [ Port 53: Standard DNS ] ──► [ DNAT Capture to AdGuard (192.168.20.5) ]

1. Drop Port 853 at the Gateway

A strict drop rule is applied on all LAN-IN interfaces for port 853:

• Action: DROP
• Protocol: TCP/UDP
• Source: ANY (Except 192.168.20.5)
• Destination: ANY:853

2. Canary Domains for Automated Browser Downgrade

To instruct browsers like Firefox and Chrome to gracefully disable opportunistic DoH without throwing connection timeout exceptions to the user, AdGuard Home returns NXDOMAIN for their standardized canary probes:

# AdGuard Home Custom Filtering Rules
# Firefox DoH canary check
||use-application-dns.net^$dnsrewrite=NXDOMAIN

# Apple iCloud Private Relay canary checks
||mask.icloud.com^$dnsrewrite=NXDOMAIN
||mask-h2.icloud.com^$dnsrewrite=NXDOMAIN

3. BGP / Firewall IP Sets for Known DoH Endpoints

For rogue clients using hardcoded IPs for DoH (bypassing canary queries), we block known public DoH endpoints (Cloudflare, Google, Quad9, NextDNS) using an automatically updated IPset feed directly on the gateway firewall rules.

3. High-Availability AdGuard Home Deployment with Docker Compose

AdGuard Home runs as a persistent service on the infrastructure node, coupled with automated TLS termination for local secure resolution.

docker-compose.yml

version: '3.8'

services:
  adguardhome:
    image: adguard/adguardhome:v0.107.54
    container_name: adguard-core
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      infra_vlan:
        ipv4_address: 192.168.20.5
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "853:853/tcp"
      - "853:853/udp"
      - "443:443/tcp"
      - "443:443/udp"
      - "3000:3000/tcp"
    volumes:
      - ./work:/opt/adguardhome/work
      - ./conf:/opt/adguardhome/conf
      - /opt/certs/live/homelab.internal:/opt/adguardhome/certs:ro
    environment:
      - TZ=America/Chicago

networks:
  infra_vlan:
    driver: macvlan
    driver_opts:
      parent: eth0.20
    ipam:
      config:
        - subnet: 192.168.20.0/24
          gateway: 192.168.20.1

Using a macvlan driver binds AdGuard directly to the physical interface tagged with VLAN 20 (eth0.20). This bypasses Docker’s iptables userland proxy, preserving client source IP attribution in query logs and eliminating NAT overhead.

4. Automated Let’s Encrypt TLS Lifecycle via Cloudflare DNS-01 API

To serve internal clients over trusted DoT/DoH within the homelab, the resolver requires a valid public TLS certificate. Because the resolver host does not expose port 80/443 to the public internet, we utilize the ACME DNS-01 challenge backed by Cloudflare API credentials.

Automated Renewal Script: /opt/scripts/renew-certs.sh

#!/usr/bin/env bash
set -euo pipefail

CERT_DIR="/opt/certs"
CF_SECRET_FILE="/opt/secrets/cloudflare_api.ini"

# Ensure secrets exist and have constrained file permissions
chmod 600 "$CF_SECRET_FILE"

echo "[$(date -Iseconds)] Initiating DNS-01 certificate renewal challenge..."

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials "$CF_SECRET_FILE" \
  --dns-cloudflare-propagation-seconds 30 \
  --non-interactive \
  --agree-tos \
  --email noc@homelab.internal \
  -d "resolver.homelab.internal" \
  --config-dir "$CERT_DIR" \
  --work-dir "$CERT_DIR/work" \
  --logs-dir "$CERT_DIR/logs"

echo "[$(date -Iseconds)] Reloading AdGuard Home container to apply fresh certificates..."
docker exec -i adguard-core /opt/adguardhome/AdGuardHome -s reload

echo "[$(date -Iseconds)] Certificates successfully synced."

Systemd Timer Configuration

We maintain the certificate cycle via an isolated systemd timer rather than a primitive cron job:

File: /etc/systemd/system/certbot-homelab.service

[Unit]
Description=Renew Internal Homelab DNS TLS Certificates
After=network-online.target

[Service]
Type=oneshot
ExecStart=/opt/scripts/renew-certs.sh
User=root
StandardOutput=journal
StandardError=journal

File: /etc/systemd/system/certbot-homelab.timer

[Unit]
Description=Bi-Weekly Run for ACME Certificate Refresh
RefuseManualStart=no
RefuseManualStop=no

[Timer]
OnCalendar=*-*-01,15 03:30:00
Persistent=true

[Install]
WantedBy=timers.target

5. Upstream Upstream Resolvers: DoQ and DNS-over-HTTPS/3

With local queries intercepted and logged, upstream resolution terminates securely to privacy-respecting authoritative resolvers using DNS-over-QUIC (DoQ). DoQ eliminates head-of-line blocking present in TCP/TLS and provides millisecond-level handshake resumption via zero-RTT (0-RTT).

Upstream Configuration (AdGuardHome.yaml)

dns:
  bind_hosts:
    - 0.0.0.0
  port: 53
  upstream_dns:
    # Primary: Cloudflare DNS-over-QUIC / DoH3
    - quic://dns.cloudflare.com
    # Secondary: Quad9 DNS-over-QUIC
    - quic://dns.quad9.net
    # Fallback: Mullvad DoT
    - tls://dns.mullvad.net

  # Bootstrap resolvers used strictly to resolve upstream hostnames
  bootstrap_dns:
    - 9.9.9.9:53
    - 1.1.1.1:53

  # Strict upstream options
  all_servers: false
  fastest_addr: true
  cache_size: 67108864 # 64MB RAM cache
  cache_ttl_min: 300
  cache_ttl_max: 86400
  edns_client_subnet:
    enabled: false # Scrub client IP geolocation leakage

Architectural Lessons & Trade-Offs

1. Macvlan vs. Host Network Interfaces

Employing macvlan grants the container an independent MAC and IP on the physical network. However, by Linux kernel design, the Docker host itself cannot communicate with the container over the macvlan interface. If local daemon processes on the host require DNS resolution from AdGuard, you must create a dedicated macvlan bridge interface on the host OS itself.

2. Upstream QUIC UDP Fragmentation

DNS-over-QUIC runs over UDP port 853. Some ISP modems or upstream transit providers handle UDP state tables poorly or fragment UDP packets larger than standard MTUs (1500). If high packet loss occurs during DoQ queries, clamping your router’s MSS or falling back to HTTP/2-based DoH guarantees delivery.

3. Failover Resilience: The “Wife Approval Factor” (WAF)

Running a single resolver container guarantees complete domestic disruption when updating software or rebooting nodes. We maintain a secondary replica on a low-power compute node (192.168.20.6), automatically kept in sync using adguardhome-sync. The UniFi gateway advertises both IPs over DHCP:

• Primary DNS: 192.168.20.5
• Secondary DNS: 192.168.20.6

Even during kernel upgrades or host reboots, DNS resolution remains sub-10ms without dropped connections.