profile picture

Cross-compiling Crystal applications - Part 2

August 11, 2024 - crystal linux compiling alpine macos

Part 2: cross-architecture is not that hard, but cross-platform is a bit more complicated.

TL;DR

Took the feedback from different folks in previous posting and integrated it into a working container version that you can find here, including instructions on how to cross-compile diverse Crystal applications to Linux (x64 and arm64) and macOS (arm64).

Update 2024-08-22:

Different architectures aren't that hard

I shared Part 1 of this exploration series on the official Crystal Forum and that triggered some great reactions and feedback! 🤗

Some suggested to use qemu-static to allow Docker leverage on binfmt support of Linux to run binaries for other architectures thanks to QEMU.

This approach means that building the container for that target architecture on the host one is going to be emulated, which might be slow. Eg. On my x86_64 machine, this will emulate aarch64 (ARM64) and generate a container for that architecture.

I used this approach in the past to build hydrofoil-crystal and found it painfully slow. Thankfully I found depot.dev that boots native Docker builders instead of emulating them, saving me lot of time in the edit-save-rebuild iteration and release cycles.

Another interesting twist to this would be cross-compile in the host (your native OS) and then leverage on the different architecture containers to only link the executable.

That approach will be faster as it leverages on your native machine to produce the instructions for the target architecture, and the only emulates the target one when linking the object files to the libraries.

Some might argue that I don't need to cross-compile all the time while developing, but if you delay testing and skip the builds as they take a long time to complete, it will make it harder for you to correct any potential problem with the approach or tools you're using, taking more of your time to troubleshoot or too late to change.

Leverage on apk for handling the packages

Another interesting note shared was to leverage on Alpine's apk to download and extract the different packages for the different architectures.

With the list of libraries from previous post, let's update the Dockerfile to leverage on apk and copy over all the .a to our container image.

Going to use /opt/multiarch-libs to isolate these from other possible setup:

Diff of changes to add libraries to Dockerfile 
+
+# install multi-arch libraries
+RUN --mount=type=cache,sharing=private,target=/var/cache/apk \
+    --mount=type=tmpfs,target=/tmp \
+    set -eux -o pipefail; \
+    supported_arch="aarch64 x86_64"; \
+    # download and extract packages for each arch
+    { \
+        cd /tmp; \
+        for target_arch in $supported_arch; do \
+            target_path="/tmp/$target_arch-apk-chroot"; \
+            mkdir -p $target_path/etc/apk; \
+            cp /etc/apk/repositories $target_path/etc/apk/; \
+            # use apk to download the specific packages
+            apk add --root $target_path --arch $target_arch --initdb --no-cache --no-scripts --allow-untrusted \
+                gc-dev \
+                libevent-static \
+                pcre2-dev \
+            ; \
+            pkg_path="/opt/multiarch-libs/$target_arch-linux-musl"; \
+            mkdir -p $pkg_path/lib; \
+            # copy the static libs
+            cp $target_path/usr/lib/*.a $pkg_path/lib; \
+        done; \
+    }

You: Wow Luis, you keep things complicated! Why can't you do a one-liner like everybody else?

Me: I'm sorry, but in 6 months I will completely forget what was the intention I had in that clever, wit one-liner.

Some benefit of this: things are downloaded inside /tmp, which is a tmpfs and avoid polluting the container image or having to remember to remove them to keep the container image small and without garbage.

If we rebuild our container (make build) and get into it (make run), we can validate our libraries are there:

$ ls -1 /opt/multiarch-libs/
aarch64-linux-musl
x86_64-linux-musl

$ ls -1 /opt/multiarch-libs/aarch64-linux-musl/lib/
libcord.a
libevent.a
libevent_core.a
libevent_extra.a
libevent_openssl.a
libevent_pthreads.a
libgc.a
libgccpp.a
libgctba.a
libpcre2-16.a
libpcre2-32.a
libpcre2-8.a
libpcre2-posix.a

Same list of libraries, and same results when compiling our simple example:

$ crystal build \
	--cross-compile --target x86_64-linux-musl \
	--static examples/hello.cr
cc hello.o -o hello  -rdynamic -static -L/usr/local/bin/../lib/crystal -lpcre2-8 -lgc -lpthread -ldl -levent

$ zig cc -target x86_64-linux-musl \
	hello.o -o hello-new \
	-L/opt/multiarch-libs/x86_64-linux-musl/lib \
	-lpcre2-8 -lgc -lpthread -ldl -levent -lunwind

$ file hello-new
hello-new: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

Nice, everything is working and easier to maintain. 🥳

Plus, it only increased the image size by 9MB:

$ docker image ls --format "{{.Repository}}\t{{.Tag}}\t{{.Size}}"
crystal-xbuild  new     688MB
crystal-xbuild  latest  679MB

Now if we need to add other libraries (Eg. OpenSSL), we can simply add those to the list and rebuild the container:

             apk add --root $target_path --arch $target_arch --initdb --no-cache --no-scripts --allow-untrusted \
                 gc-dev \
                 libevent-static \
+                openssl-libs-static \
                 pcre2-dev \
             ; \
             pkg_path="/opt/multiarch-libs/$target_arch-linux-musl"; \

Overall, here is the list of packages that I end extracting the static libraries from:

You might be wondering why I'm also adding certain -dev versions of packages when the -static one provides our static libraries, please keep reading 😁

You not only need static libraries, but pkg-config files too

For certain libraries like OpenSSL, Crystal needs to determine the version available in order to correctly activate certain functionality. For this, it leverages on pkg-config and queries the .pc catalog to determine the version and possible any additional libraries that might be required when linking. See @[Link] annotation.

Since Crystal will attempt to execute pkg-config, it means it will use what is known by our host about these libraries. These might differ, or might be different.

Way more libraries than the ones we copied before. If for example I was compiling in my host machine, Crystal might grab different version information of OpenSSL (see here for the code)

Trying this on my macOS host:

$ pkg-config --silence-errors --modversion libssl || printf %s 0.0.0
3.3.1

Versus the version in the container:

$ pkg-config --silence-errors --modversion libssl || printf %s 0.0.0
3.1.5

Different versions, and possible different outcomes. Take the previous scenario (compile in host, link inside the container) and things could possibly not compile.

To avoid issues, we can tell pkg-config where to look for the exact definitions of our packages, but first we need to include them in our image.

These package definitions (.pc files) are normally found within -dev variant of the packages. So let's go back to our Dockerfile and include those variants and copy the respective files.

Diff of changes to add dev libraries and copy the pkg-config files 
             # use apk to download the specific packages
             apk add --root $target_path --arch $target_arch --initdb --no-cache --no-scripts --allow-untrusted \
                 gc-dev \
+                gmp-dev \
+                libevent-dev \
                 libevent-static \
+                openssl-dev \
+                openssl-libs-static \
                 pcre2-dev \
+                sqlite-dev \
+                sqlite-static \
+                yaml-dev \
+                yaml-static \
+                zlib-dev \
+                zlib-static \
             ; \
             pkg_path="/opt/multiarch-libs/$target_arch-linux-musl"; \
-            mkdir -p $pkg_path/lib; \
-            # copy the static libs
-            cp $target_path/usr/lib/*.a $pkg_path/lib; \
+            mkdir -p $pkg_path/lib/pkgconfig; \
+            # copy the static libs & .pc files
+            cp $target_path/usr/lib/*.a $pkg_path/lib/; \
+            cp $target_path/usr/lib/pkgconfig/*.pc $pkg_path/lib/pkgconfig/; \
         done; \
     }

Let's rebuild the image (make build) and validate we have the pkg-config information for our libraries:

$ export PKG_CONFIG_LIBDIR=/opt/multiarch-libs/x86_64-linux-musl/lib/pkgconfig

$ pkg-config --list-package-names
bdw-gc
form
formw
gmp
gmpxx
libcrypto
libedit
libevent
libevent_core
libevent_extra
libevent_openssl
libevent_pthreads
libpcre2-16
libpcre2-32
libpcre2-8
libpcre2-posix
libssl
menu
menuw
ncurses++
ncurses++w
ncurses
ncursesw
openssl
panel
panelw
sqlite3
yaml-0.1
zlib

We have the libraries we wanted, but got some others that we did not request?!

This is because apk will install all the dependencies your packages depends on and since pcre2-dev depends on libedit-dev, which depends on ncurses-dev... you get them all. 🫤

No harm for the time being, since we are not linking them, we don't need to worry about them. The only downside is that it will only increase the size of our final container image.

With all these libraries, we can now cross-compile more complex applications like HTTP servers or clients, or use the crystal-sqlite3 DB adapter.

macOS does not have apk...

All the work we did before was only for Linux, but if we want something truly cross-platform, we need to consider other platforms, like macOS.

Note that while Windows support in Crystal is getting better, does not yet provide an stable playground to my expectations. You're free to take whatever I mention here, adapt and explore to support it.

Having to deal with another platform means I also need to figure out how I'm going to get packages for it. I cannot leverage on apk for macOS, but I could use Homebrew packages.

The downside is that Homebrew is written in Ruby and attempting something like I did before with apk might be too complicated to setup, specially since we will be trying from within a Linux installation to run something that targets macOS.

This is why in the past I wrote a Ruby/Rake script that downloaded specific packages from Alpine and Homebrew repositories and extracted the necessary files.

Maintaining the list of files to download was a lot of work, so perhaps is a good moment to automate that too.

Extracting list of packages from Homebrew's API

Homebrew provides a JSON API that contains all the metadata of the formulas available for you to install.

Let's download that JSON endpoint and peek into the contents to see if I can automate the list of package. This time, using Crystal!

Downloaded formula.json and started to look for the packages we need 🤔

$ curl -LO https://formulae.brew.sh/api/formula.json

Each formula is very verbose, but here are some of fields I found interesting:

Snippet of formula.json focused on bdw-gc (libgc) 
[
    ...
    {
      "name": "bdw-gc",
      ...
      "aliases": [
        "boehmgc",
        "libgc"
      ],
      ...
      "versions": {
        "stable": "8.2.6",
        "head": "HEAD",
        "bottle": true
      },
      ...
      "bottle": {
        "stable": {
          ...
          "files": {
            ...
            "arm64_monterey": {
              "cellar": ":any",
              "url": "https://ghcr.io/v2/homebrew/core/bdw-gc/blobs/sha256:d98f35081558a6411f47913a4da75a1d72449e08534ea27e113f3872b52654b2",
              "sha256": "d98f35081558a6411f47913a4da75a1d72449e08534ea27e113f3872b52654b2"
            },
            ...
            "monterey": {
              "cellar": ":any",
              "url": "https://ghcr.io/v2/homebrew/core/bdw-gc/blobs/sha256:9f2c45bbb24805adaec4a3be2cbedad416ec8ff46a8ea558e1e11c0b7cec3ced",
              "sha256": "9f2c45bbb24805adaec4a3be2cbedad416ec8ff46a8ea558e1e11c0b7cec3ced"
            },
          }
        }
      },
      ...
    },
    ...
]

Going to focus on arm64_monterey (Monterey) for compatibility, as is the oldest version of macOS that hasn't reached EOL and will help to distribute my applications to old installations.

Let's start simple:

require "json"

struct BrewFormula
  include JSON::Serializable

  getter name : String
  getter aliases : Array(String)
end

formulas = File.open("formula.json", "r") do |io|
  Array(BrewFormula).from_json(io)
end

p! formulas.size

And running that program, shows us the number of formulas available:

$ crystal run scripts/build-homebrew-download-list.cr
formulas.size # => 7085

Now is time we start filtering that list!

So:

  1. A simple CLI tool like apk that downloads a given list of packages for a platform
  2. It downloads the packages from GitHub artifacts and extract them
  3. Then copies the .a and .pc of our packages into our final container image

Narrator: picture here a cooking show, where Luis breaks some eggs, mix them with flour and other stuff, then put everything into the oven and show us a finished, beautiful cake.

Terrible cooking analogy, but we end with homebrew-downloader.cr script:

$ crystal run scripts/homebrew-downloader.cr \
  <output_dir> <package1> <package2> ...

Note that we can use package names or aliases, so instead of requesting bdw-gc (Boehm GC), we can use libgc as the alias.

Here is a list of the packages we will be downloading and the matching apk counterpart:

$ crystal run scripts/homebrew-downloader.cr -- \
    /tmp/arm64-homebrew-root \
    gmp \
    libevent \
    libgc \
    libyaml \
    openssl \
    pcre2 \
    sqlite \
    zlib \
  ;

We will end with the following files being extracted:

$ tree /tmp/arm64-homebrew-root
/tmp/arm64-homebrew-root
└── lib
    ├── libcord.a
    ├── libcrypto.a
    ├── libevent.a
    ├── libevent_core.a
    ├── libevent_extra.a
    ├── libgc.a
    ├── libgccpp.a
    ├── libgctba.a
    ├── libgmp.a
    ├── libgmpxx.a
    ├── libpcre2-16.a
    ├── libpcre2-32.a
    ├── libpcre2-8.a
    ├── libpcre2-posix.a
    ├── libsqlite3.a
    ├── libssl.a
    ├── libyaml.a
    ├── libz.a
    └── pkgconfig
        ├── bdw-gc.pc
        ├── gmp.pc
        ├── gmpxx.pc
        ├── libcrypto.pc
        ├── libpcre2-16.pc
        ├── libpcre2-32.pc
        ├── libpcre2-8.pc
        ├── libpcre2-posix.pc
        ├── libssl.pc
        └── openssl.pc

Now, let's modify our Dockerfile to download and include these files:

Diff of changes to add libraries to Dockerfile 
+
+# ---
+# macOS
+
+# install macOS dependencies in separate target
+FROM base AS macos-packages
+COPY ./scripts/homebrew-downloader.cr /homebrew-downloader.cr
+
+RUN --mount=type=cache,sharing=private,target=/var/cache/apk \
+    --mount=type=tmpfs,target=/tmp \
+    set -eux -o pipefail; \
+    # macOS (Monterey), supports only Apple Silicon (aarch64/arm64)
+    { \
+        pkg_path="/opt/multiarch-libs/aarch64-apple-darwin"; \
+        mkdir -p $pkg_path/lib/pkgconfig; \
+        # run homebrew-downloader
+        crystal run /homebrew-downloader.cr -- \
+            $pkg_path \
+            gmp \
+            libevent \
+            libgc \
+            libyaml \
+            openssl \
+            pcre2 \
+            sqlite \
+            zlib \
+        ; \
+    }
+
+# copy macOS dependencies back into `base`
+FROM base
+COPY --from=macos-packages --chmod=0444 /opt/multiarch-libs/aarch64-apple-darwin /opt/multiarch-libs/aarch64-apple-darwin

We use a separate target to avoid temporary build artifacts from crystal run polluting our final container image.

Let's test to cross-compile an application for macOS!

$ crystal build \
  --cross-compile --target aarch64-apple-darwin \
  --static examples/hello.cr
cc hello.o -o hello  -rdynamic -static -L/usr/local/bin/../lib/crystal -lpcre2-8 -lgc -lpthread -ldl -levent -liconv

$ zig cc -target aarch64-macos-none \
  hello.o -o hello-mac \
  -L/opt/multiarch-libs/aarch64-apple-darwin/lib \
  -lpcre2-8 -lgc -lpthread -ldl -levent -liconv

Sadly this fails, as it indicates we are missing iconv, which is part of macOS:

error: unable to find dynamic system library 'iconv' using strategy 'paths_first'. searched paths:
  /opt/multiarch-libs/aarch64-apple-darwin/lib/libiconv.tbd
  /opt/multiarch-libs/aarch64-apple-darwin/lib/libiconv.dylib
  /opt/multiarch-libs/aarch64-apple-darwin/lib/libiconv.so
  /opt/multiarch-libs/aarch64-apple-darwin/lib/libiconv.a

We also need macOS SDK 🫠

Is not enough having all the static libraries we already put in place, but we need certain system libraries in order to link a macOS executable.

For that, we can rely on existing macOS SDK. There are several ways to obtain them.

Diff of changes to add macOS SDK to Dockerfile 
+
+# install macOS SDK
+RUN --mount=type=cache,sharing=private,target=/var/cache/apk \
+    --mount=type=tmpfs,target=/tmp \
+    set -eux -o pipefail; \
+    { \
+        cd /tmp; \
+        export \
+            MACOS_SDK_VERSION=12.3 \
+            MACOS_SDK_MAJOR_VERSION=12 \
+            MACOS_SDK_SHA256=3abd261ceb483c44295a6623fdffe5d44fc4ac2c872526576ec5ab5ad0f6e26c \
+        ; \
+        wget -q -O sdk.tar.xz https://github.com/joseluisq/macosx-sdks/releases/download/${MACOS_SDK_VERSION}/MacOSX${MACOS_SDK_VERSION}.sdk.tar.xz; \
+        echo "${MACOS_SDK_SHA256} *sdk.tar.xz" | sha256sum -c - >/dev/null 2>&1; \
+        tar -C /opt/multiarch-libs -xf sdk.tar.xz; \
+        rm sdk.tar.xz; \
+        # symlink to latest version
+        ln -nfs /opt/multiarch-libs/MacOSX${MACOS_SDK_VERSION}.sdk /opt/multiarch-libs/MacOSX${MACOS_SDK_MAJOR_VERSION}.sdk; \
+    }

This will download latest macOS SDK compatible with Monterey and make sure is available to us.

With that in place, let's try to link again, but this time providing adding the path to macOS SDK:

$ zig cc -target aarch64-macos-none \
  hello.o -o hello-mac \
  -L/opt/multiarch-libs/aarch64-apple-darwin/lib \
  -L/opt/multiarch-libs/MacOSX12.sdk/usr/lib \
  -lpcre2-8 -lgc -lpthread -ldl -levent -liconv

$ file hello-mac
hello-mac: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS|PIE|HAS_TLV_DESCRIPTORS>

It linked! 🎉

Let's now run that executable in my host (macOS Sonoma) to confirm it works:

$ uname -sm
Darwin arm64

$ otool -L hello-mac
hello-mac:
        /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

$ ./hello-mac
Hello World!

The file links correctly to macOS system libraries, and it works! 🥳

Now, let's try to cross-compile a more complex example (an HTTP::Server). This time, as Crystal needs to detect the version of OpenSSL, we need to provide the directory for pkg-config:

$ export PKG_CONFIG_LIBDIR=/opt/multiarch-libs/aarch64-apple-darwin/lib/pkgconfig

$ crystal build \
  --cross-compile --target aarch64-apple-darwin \
  --static examples/server.cr
cc server.o -o server  -rdynamic -static -L/usr/local/bin/../lib/crystal -lz `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libssl || printf %s '-lssl -lcrypto'` `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libcrypto || printf %s '-lcrypto'` -lpcre2-8 -L@@HOMEBREW_CELLAR@@/bdw-gc/8.2.6/lib -lgc -lpthread -levent -liconv

A very mouthful command line, but we only care about the list of libraries: -lz -lssl -lcrypto -lpcre2-8 -lgc -lpthread -levent -liconv

Let's link it now:

$ zig cc -target aarch64-macos-none \
  server.o -o server-mac \
  -L/opt/multiarch-libs/aarch64-apple-darwin/lib \
  -L/opt/multiarch-libs/MacOSX12.sdk/usr/lib \
  -lz -lssl -lcrypto -lpcre2-8 -lgc -lpthread -levent -liconv

And we can run it on our macOS host:

$ ./server-mac
Listening on http://127.0.0.1:8080

$ curl -i http://localhost:8080/foo/bar
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/plain
Content-Length: 26

Hello world, got /foo/bar!

Lot of manual commands...

I know, I know, lot of typing, lot to remember, better to automate it, no?

Let's add a simple script that cross-compile a given binary for the supported targets and that handles all the variables and details for us: xbuild.sh

Add xbuild.sh to Dockerfile 
diff --git a/Dockerfile b/Dockerfile
index fbe1c13..f0f0add 100644
+
+# copy xbuild helper
+COPY ./scripts/xbuild.sh /usr/local/bin/xbuild

And now rebuild the container image with make build.

This simple script will allow you to cross-compile to the targets we added before:

$ xbuild
Usage: /usr/local/bin/xbuild <filename> <executable_name> <target_platform>

$ xbuild examples/hello.cr hello-mac aarch64-apple-darwin
Compiling 'build/aarch64-apple-darwin/hello-mac' ('examples/hello.cr')...
Linking with: -lpcre2-8 -lgc -lpthread -levent -liconv
Done.

Your new binary is in build/ directory, to avoid clashing with any existing executable in bin.

Works, but it could be better

I promised to build macOS binaries, so that works, but still, this could be better.

One example could be this part of shards as a subcommand (Eg. shards xbuild), working natively without the need of a container image.

In order to make that happen, Shards needs to forward all the arguments to the subcommand, something that this PR #631 aims to solve.

Enough for today. Let's see if next time we can make all these things natively without the need of a container image.

Enjoy! ❤️