Every public request to one of my sites used to go through a Cloudflare tunnel. I had a cloudflared deployment in my cluster, some routing rules, and everything just worked. And it was free! But that bugged me. TANSTAAFL right? The whole point of selfhosting is to not rely on big cloud company, and there are few bigger than Cloudflare. And Cloudflare’s tunnels decrypt every single packet, so they see everything you do. I really don’t like that.

So I’ve replaced Cloudflare tunnels with Erwan Leboucher’s towonel, a selfhosted tunnel implementation. I still have my DNS on Cloudflare, but no data is moving through them any more.

My implementation plan:

  1. Stand up a VPS with a tunnel hub and get one hostname through it end to end
  2. Solve DNS so hostnames can be migrated one at a time
  3. Migrate everything, keeping cloudflared as a rollback
  4. Rebuild the WAF and bot protection I was about to lose
  5. Delete cloudflared
internet ──▶ heimdall (a small OVH box)
                ├─ Caddy L4 edge     ACME TLS for the hub API,
                │                    SNI :443 passthrough ─▶ towonel-hub (PROXY v2)
                ├─ towonel-hub
                ├─ crowdsec
                └─ observability
                        │
                        │ tunnel
                        ▼
 cluster ── towonel-agent ──[PROXY v2]──▶ envoy-towonel (a dedicated Gateway)
                                            ├─ Coraza WAF
                                            ├─ rate limits / Anubis
                                            └─ app HTTPRoutes

A hub runs on a cheap public VPS (~$5/month), an agent runs in the cluster, and the agent dials out to the hub. So there’s still no inbound port open at my house and my IP isn’t exposed. Same model as cloudflared, just my own infra.

The VPS is the only non-Kubernetes thing in my repo right now. Its a stack of Docker Compose files managed by doco-cd. It essentially does for Docker Compose what FluxCD does for Kubernetes, including pulling in secrets from Bitwarden.

The one design decision worth calling out: Caddy does L4 SNI passthrough, not TLS termination. It reads the SNI field, forwards the still-encrypted bytes to the hub, and never sees any plaintext. Only the hub’s own API hostname gets a cert from Caddy. Everything else is terminated inside the cluster with my certs, which was the whole point.

Getting the Client IP Through

I built the whole thing in a day (though it is largely cribbed from Erwan’s repo) and it worked. Except, every request in my logs appeared to come from the agent pod. Which makes rate limiting, geo-blocking, and abuse investigation pretty difficult.

The fix is PROXY protocl, where the proxy prepends the original source address before the real bytes. Townel supports it per service, but note that proxy_protocl is a string enum, not a bool.

services:
  - hostname: "*.example.com"
    origin: envoy-townel.networking.svc.cluster.local:443
    proxy_protocol: "v2"

The bigger problem is that a listener speaking PROXY protocol only speaks that protocol. Send it a plain TLS handshake and it drops the connection. My gatway was also serving LAN traffic, which obviously doesn’t arrive wrapped in a PROXY header.

So I gave the tunnel its own gateway. envoy-towoenl is ClusterIP-only, but only the agent talks to it anyways. Then gave it its own policy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: envoy-towonel
  namespace: networking
spec:
  proxyProtocol: {}
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: envoy-towonel

And then excluded it from the cluster-wide policy, which otherwise re-applies the LAN settings:

  targetSelectors:
    - group: gateway.networking.k8s.io
      kind: Gateway
      matchExpressions:
        - { key: network, operator: NotIn, values: [towonel] }

Having a dedicated gateway made everything smoother. Now I can add the same app on envoy-internal and envoy-towonel and access it from home while bypassing the tunnel entirely.

External-DNS

My cloudflare external-dns instance ran with --cloudflare-proxied --default-targets=external.wynning.tech --force-default-targets. That means it would automatically point all my records at the cloudflare tunnel. I needed some records pointed directly to the VPS though. So I had to change my it to filter only the towonel network:

  extraArgs:
    - --gateway-label-filter=network in (towonel)
  sources:
    - gateway-httproute
    - crd
  txtOwnerId: towonel
  txtPrefix: k8s-tw.

Then I just added the hostname or wildcards to the towonel-agent’s service list:

  - hostname: "jameswynn.com"
    origin: envoy-towonel.networking.svc.cluster.local:443
    proxy_protocol: "v2"
  - hostname: "*.jameswynn.com"
    origin: envoy-towonel.networking.svc.cluster.local:443
    proxy_protocol: "v2"

Rebuilding the WAF

Cloudflare provides a pretty robust WAF and I knew I needed a replacement for that. I used Coraza, a WASM ModSecurity-compatible engine that runs as an Envoy extension, plus the OWASP Core Rule Set. One EnvoyExtensionPolicy attached to the envoy-towonel gateway and all routes are covered.

I ran it in detection-only for a couple days with a Grafana dashboard of detections by rule family, added some rules for things that were caught, and then flipped it to blocking. I still have to regularly analyze the logs to find apps that get caught in it though.

A few issues that came up:

  • Nextcloud CalDAV sync is cross-site scripting. Calendar data is XML, the XSS rules see the markup and score it. And .ics files use CRLF line folding, which the header-injection rules read as HTTP response splitting.
  • git is a null byte injection. Smart-HTTP sends binary packfiles, and Coraza parses the body as if it were form input. Every git push got a 403.
  • Navidrome’s Subsonic API is SQL injection. The auth token is an MD5 hash and libinjection reads some hex strings as SQL.
  • Web Push notifications are, statistically, an attack. RFC 8291 bodies are AES ciphertext with Content-Encoding: aes128gcm and no content type. Random bytes reliably contain null bytes and CRLF pairs. My ntfy pushes were scoring 15-50 against a threshold of 5.

Each one gets a tightly -scoped exclusion in the rule file that gets executed before the CRS rules. Here’s an example:

# Nextcloud CalDAV/WebDAV: XML markup trips the XSS family (941xxx), the sync
# bodies aren't in the allowed content-type set (920420), and CRLF-folded .ics
# lines look like header injection (921120/921150/921160).
- 'SecRule REQUEST_URI "@rx ^/(remote|public)\.php/(web)?dav" "id:900310,phase:1,pass,nolog,t:none,ctl:ruleRemoveById=941000-941999,ctl:ruleRemoveById=920420,ctl:ruleRemoveById=921120,ctl:ruleRemoveById=921150,ctl:ruleRemoveById=921160"'

Scrapers

The WAF handles injection, not AI scrapers, which started hammering my Forgejo instance. I already had Anubis but I had to roll it out a bit further. It is a proof-of-work gate with the cat-girl that you see everywhere these days. So Anubis, then rate limits on anonyous traffic, and CrowdSec on the VPS to drop repeat offenders.

Unfortunately because its an L4 passthrough there were no HTTP logs on the VPS, so CrowdSec needs to tail Envoy’s access logs via loki over tailscale.

Conclusion

With all of this I took on a pretty big new piece of infra maintenance. The WAF alone adds some regular work to tweak, and I probably would still get taken down by a real DDoS attack. I’m not too concerned about that though. Maybe OVH could handle it? Hopefully I’ll never find out.

If you’d like to check out the source here are the relevant links: