Build Tools Wars

The Quest for the Ultimate Build Tool

Long gone are the days when we had to memorize every single step of executing a given task, whether big or small. I’m always trying to find ways to automate tasks and free my human memory for more important things instead of having to remember how to execute certain tasks. Learn, automate, and move on—that’s my philosophy!

I’ve been relying on Makefiles and the traditional Dockerfiles for a long time. After all, they are battle-tested and work well. But… there’s always a “but”! When you start to scale and standardize how you do things, these two don’t scale with you. If you have multiple projects that build the same way with only minor differences, sooner or later, you’ll update something in one and wish it would magically propagate to all your other projects.

Moreover, at some point, you’ll want to ensure that your builds, tests, and tasks run exactly the same way, regardless of whether it’s on your local machine or in a CI environment. Sure, it’s not a big deal when you have just two or three projects. But start scaling beyond a handful, and it becomes a real annoyance.

This is where Taskfile, Earthfile (Earthly), and Dagger come into play. All of them introduce a solid dose of steroids to your Makefiles, Dockerfiles, or even both! But which one should I use? And how should I use it?

My Requirements

To make a fair assessment, here are my key requirements for the ideal tool:

• Task Reusability: I should be able to define tasks and reuse them across multiple repositories or projects inside a monorepo.
• Container Build Reusability: The tool should allow me to define container builds and reuse those definitions across multiple projects.
• Environment Consistency: Builds should behave the same way on both my local machine and any CI environment.

To put these tools to the test, I used my most recent project (at the time of writing): aeye. This is a Python project I built to automate one of my YouTube/Twitch channels, The Arctic Skies. It trains and runs an AI model that detects the Northern Lights from the camera video feed.

The tasks I need it to perform are:

• Build a container image.
• Run the container image with some variables set for testing.
• Run the same container image but with a shell instead, so I can troubleshoot issues within the image.

Taskfile: Not Quite Enough

Let’s start with the simplest option, and also the only one on this list I did not test at all: Taskfile.

There is a straightforward reason why I did not test Taskfile—it only replaces Makefiles. While it provides a nice YAML-based approach to writing tasks and encourages reusability, I wanted to stop duplication in both Makefiles and Dockerfiles.

If all you want is a Makefile on steroids with a cleaner YAML syntax, Taskfile is worth checking out. But since it doesn’t fully meet my requirements, I kept exploring.

Dagger: The Cool but Unusual Option

Dagger looks really promising. It can replace your Dockerfiles, and it claims to replace Makefiles as well. However, after some initial reading, I had doubts about the latter claim. So, I rolled up my sleeves and tested it out.

I won’t go over every feature—if you want that, head over to dagger.io and check it out yourself. Instead, let’s focus on what makes Dagger unique: you define your builds using actual code! You can use Go, Python, TypeScript, and more to define your build pipeline.

Even though it works really well, is super fast, and quite cool to use, Dagger forces you to write in a weird coding style that I found hard to read and mentally parse. Also, all the Go code runs in a container via the Dagger engine. There is no native/local execution mode, and after speaking with some devs, they didn’t seem keen on adding such a feature.

Overall, I liked Dagger, but the forced code style and lack of local execution made me hesitant.

Earthly: The Best of Both Worlds

Earthly is a convenient merge between a Dockerfile and a Makefile, and it’s a powerful tool. It provides a syntax that feels familiar, combined with a build engine similar to Dagger.

Earthly brings Makefile-like functionalities on steroids—allowing you to import targets and functions from other Earthfiles, whether in the same repository or even in a separate Git repository.

It was a breeze to learn, the syntax is familiar, and reusability is built-in, but I wish you could just import an Earthfile with two lines like this:

VERSION 0.8.0
IMPORT .. as common

Instead, you have to create functions in a common Earthfile as exemplified in my dry repository and then call them in project-specific targets as I’ve done in my aeye project. This adds a bit of complexity but is still manageable.

The Uncertain Future of Earthly

Unfortunately, Earthly’s growth seems to be outpacing its ability to maintain Earthfiles, and its future is a bit uncertain at the time of writing.

I’m not one to dip into tech politics, but I wonder if something similar to OpenTofu’s fork from Terraform could happen here. Instead of Earthly potentially rug-pulling its users, they could donate the Earthfile codebase to the Linux Foundation or Cloud Native Computing Foundation. This would allow multiple companies to drive the project under proper governance.

Final Thoughts

I love Go, and I seriously considered combining Dagger with Taskfile or Earthly. But ultimately, I would still need two tools to do the job, and I really didn’t appreciate the long function signatures and notations that Dagger forces you to use.

Even though Earthly’s future is uncertain, for now, I’m sticking with it because it solves and abstracts all my problems with a single tool.

TL;DR

Right now, Earthly wins, but I’ll be keeping an eye on how things evolve! 🚀