Testing out headscale for a homelab setup

Instead of just exposing a single machine to the internet (see previous TIL), I wanted this time to quickly test headscale and see how it operates for a homelab scenario to replace Tailscale.

▶️ I also made a recording of this that you can watch on YouTube here.

---

1. Run headscale in a controlled environment (Container)

Since I didn't had experience with headscale before, looked at their instructions and decided to test their container image using Docker Compose:

# compose.yaml
---
version: "3.9"

services:
  headscale:
    image: ghcr.io/juanfont/headscale:0.19
    command: headscale serve

    ports:
      # Listen on virtual bridge (virbr0)
      - "192.168.122.1:8080:8080"

    volumes:
      - ./config/headscale.yaml:/etc/headscale/config.yaml
      - headscale:/var/lib/headscale

volumes:
  headscale:
    driver: local

And created a local headscale.yaml configuration file following the docs config-example.yaml:

---
server_url: http://192.168.122.1:8080
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
private_key_path: /var/lib/headscale/private.key
noise:
  private_key_path: /var/lib/headscale/noise_private.key

ip_prefixes:
  # - fd7a:115c:a1e0::/48
  - 100.64.0.0/10

# Disables the automatic check for headscale updates on startup
disable_check_updates: true

db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite

log:
  format: text
  level: info

dns_config:
  override_local_dns: true
  nameservers:
    - 1.1.1.1
    - 1.0.0.1
    # - 2606:4700:4700::1111
    # - 2606:4700:4700::1001

  magic_dns: true
  base_domain: luislavena.info

logtail:
  enabled: false

Notes:

1. Used 192.168.122.1 for the server address, as it corresponds to the host IP in the KVM/QEMU virtual bridge
2. Disabled IPv6 for the purpose of testing
3. Opted for SQLite3 as database for future backup options using Litestream
4. Pointed all generated files to /var/lib/headscale, which was a mounted volume for the container.

2. Install Tailscale in the client machines (VMs)

Since I was using Alpine Linux, this required to enable the community repository in /etc/apk/repositories:

 https://dl-cdn.alpinelinux.org/alpine/v3.17/main
+https://dl-cdn.alpinelinux.org/alpine/v3.17/community

Install the package and ensure the service is running:

$ apk add tailscale

$ service tailscale start

3. Adding a shell to headscale container to easy management

Headscale official image lacks any shell (Eg. bash/ash), while I can use docker compose exec to run one-off commands against headscale executable, it will be more practical to be able to shell into the container and run them more easily. This can also help later on when thinking on cloud deployment to something like Fly.io and the flyctl ssh console.

Decided to adjust the Docker Compose configuration and build a new image locally:

 services:
   headscale:
-    image: ghcr.io/juanfont/headscale:0.19
+    build: .
     command: headscale serve

And created a new Dockerfile:

FROM alpine:3.17.1

# ---
# upgrade system and installed dependencies for security patches
RUN --mount=type=cache,sharing=private,target=/var/cache/apk \
    set -eux; \
    apk upgrade

# ---
# copy headscale
RUN --mount=type=cache,target=/var/cache/apk \
    --mount=type=tmpfs,target=/tmp \
    set -eux; \
    cd /tmp; \
    { \
        export \
            HEADSCALE_VERSION=0.19.0 \
            HEADSCALE_SHA256=76e62be5f8a82763995903d413fa71c57143ea0b9c21d376be66793fdb6e993a; \
        wget -q -O headscale https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64; \
        echo "${HEADSCALE_SHA256} *headscale" | sha256sum -c - >/dev/null 2>&1; \
        chmod +x headscale; \
        mv headscale /usr/local/bin/; \
    }; \
    # smoke tests
    [ "$(command -v headscale)" = '/usr/local/bin/headscale' ]; \
    headscale version

(I know, the file is too verbose), so to summarize it:

1. Uses Alpine Linux as base
2. Makes sure all security packages are installed
3. Download an specific version of headscale and verify it, then verify is working.

Now make sure to build the new image and restart the container:

$ docker compose down

$ docker compose build

$ docker compose up

Now is possible to start a new session inside the container:

$ docker compose exec headscale sh -i
/ #

And we can now create the new user (mesh network):

$ headscale users create homelab

3. Connect client VM and register nodes in control plane

Inside each client VM:

$ tailscale up --login-server http://192.168.122.1:8080

Follow the instructions and copy the registration command to execute inside the headscale shell session:

$ headscale --user homelab nodes register --key <KEY>

4. Test network is working

Since we have MagicDNS enabled, you can access each of the registered nodes/machines by their name:

$ ping node1.homelab.luislavena.info

If is the first time, it might take a few seconds to establish the connection.

5. What's next?

With basic headscale operational, next step will be see how I could backup it's configuration (SQLite3 database) in case of disaster.

Then, a proper, public deployment so I could roll this to my real devices 😊