Deploying headscale to Fly.io for homelab setup

February 19, 2023 - wireguard tailscale headscale homelab fly

This time I took my local headscale setup and deployed to Fly.io, to allow nodes from different locations to connect into my homelab mesh network.

This builds on top of work done in previous TIL, but moves away from local testing using Docker and some VMs to real internetz deployment.

▶️ Also made a recording of this that you can watch on YouTube here.

1. Adjusting existing configuration

Decided to do some adjustments to headscale.yml configuration file in order to make it easier for me to configure Fly's mount volumes:

-private_key_path: /var/lib/headscale/private.key
+private_key_path: /data/private.key

-  private_key_path: /var/lib/headscale/noise_private.key
+  private_key_path: /data/noise_private.key

-db_path: /var/lib/headscale/headscale.sqlite3
+db_path: /data/headscale.sqlite3

2. Create an application in Fly

Used --generate-name for testing, but you should give a good, memorable name for your application 😉:

$ flyctl apps create --generate-name 
New app created: dark-fire-450

And create the volume that we will use to store headscale's database:

$ flyctl volumes create --app dark-fire-450 --region cdg --size 1 headscale_data
        ID: vol_2n0l9vllx58v635d
      Name: headscale_data
       App: dark-fire-450
    Region: cdg
      Zone: 0e8c
   Size GB: 1
 Encrypted: true
Created at: 19 Feb 23 12:23 UTC

3. Adjust configuration

The flyctl apps create command didn't generate a configuration file, so we are going to manually create it:

app = "dark-fire-450"
kill_signal = "SIGINT"
kill_timeout = 5

[metrics]
port = 9090
path = "/metrics"

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 8080
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[mounts]
source = "headscale_data"
destination = "/data"

Notice app indicates the name of the application we just created (dark-fire-450) and source mountpoint uses the name of the volume we created after.

4. Deploying and updating our configuration

We are ready to deploy our new application:

$ flyctl deploy --region cdg
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-old-fire-5640 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
...
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Logs: https://fly.io/apps/dark-fire-450/monitoring

 1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 critical]

Note that used cdg for this application since is the closest region to my location, but you could use other. See flyctl platform regions for alternatives.

After a few seconds, we should be able to SSH into our headscale deployment

$ flyctl ssh console 
Connecting to fdaa:0:5fde:a7b:5b66:5:4cd:2... complete

And we can create our homelab network and keys to use with our nodes:

/ # headscale users create homelab
User created

/ # headscale --user homelab preauthkeys create --reusable --expiration 24h
df72670243f0f635bc2b5d73e5a42f1c40564d9c954dfd78

This time I used preauthorized keys so I could apply the same key to multiple nodes without having to approve each node manually.

Then, tested against a remote servers and connected them to the mesh network, using the same key:

$ tailscale up --login-server https://dark-fire-450.fly.dev:443 --authkey df72670243f0f635bc2b5d73e5a42f1c40564d9c954dfd78

5. What's next?

Now we have some data out there, what would be our backup strategy? How could we avoid losing all the configuration of our nodes and mesh network?