Deduplicating git worktrees on EXT4 with jdupes

A git worktree workflow eats a lot more disk than you’d expect. On the remote box I run my coding agents on , ten worktrees took a 28 GB volume down to 687 MB free in about a day. Here’s one way to limit that.

Ten worktrees, 9.9 GB#

Every agent gets its own worktree, which is the whole point: two agents editing the same working directory trip over each other instantly. What I hadn’t thought through is what a worktree costs.

$ du -xh -d2 ~/.t3/worktrees | sort -rh | head -4
9.7G	/home/dev/.t3/worktrees
5.9G	/home/dev/.t3/worktrees/ohdear.app
3.7G	/home/dev/.t3/worktrees/ma.ttias.be
1.9G	/home/dev/.t3/worktrees/ma.ttias.be/t3code-c207f00a

The checkout itself is cheap. Git shares one .git across all worktrees, so the source files are a rounding error. The expensive part is what my setup script does after the checkout: composer install and npm ci.

On the Oh Dear app that’s about 530 MB of node_modules and 350 MB of vendor. Call it 880 MB per worktree. Ten worktrees, nine gigabytes, and almost all of it byte-for-byte identical because most of those branches share a lockfile.

The right answer to “I need ten near-identical copies of a directory” is a reflink. On a copy-on-write filesystem, cp --reflink makes a clone that shares the underlying blocks with the original: nothing gets copied up front, and when one side writes, only the changed blocks get their own copy. That’s the exact shape of this problem.

So I tried it.

$ findmnt -no FSTYPE,SOURCE /
ext4 /dev/vda1

$ cp --reflink=always a.bin b.bin
cp: failed to clone 'b.bin' from 'a.bin': Operation not supported

Both files are on the same filesystem. There’s no flag to fix this. ext4 has no copy-on-write, so it has no reflinks, and a standard Ubuntu cloud image gives you ext4 on the root volume. Reflinks need btrfs, XFS formatted with reflink=1, or ZFS.

I could reformat the root filesystem of a live box for a dev-convenience win. I’m not going to.

What ext4 does have is hardlinks. Two names pointing at one inode, one copy of the data on disk. jdupes finds byte-identical files and replaces the duplicates with hardlinks.

The whole idea, with two 8 MB files:

$ ls -li a.bin b.bin
761293 -rw-r--r-- 1 dev dev 8388608 Jul 24 08:08 a.bin
761296 -rw-r--r-- 1 dev dev 8388608 Jul 24 08:08 b.bin
$ du -sh .
17M	.

$ jdupes -L -r .

$ ls -li a.bin b.bin
761293 -rw-r--r-- 2 dev dev 8388608 Jul 24 08:08 a.bin
761293 -rw-r--r-- 2 dev dev 8388608 Jul 24 08:08 b.bin
$ du -sh .
8.1M	.

Same inode, link count 2, half the disk. -L does the linking, -r recurses. jdupes compares candidates byte for byte rather than trusting a hash, so it only links on real identity.

This isn’t copy-on-write. Hardlinked files don’t diverge on write: write into one and you’ve written into all of them. That would be a disaster for source files.

It’s fine for dependencies because of how package managers write. npm and Composer don’t edit files in place, they write a new file and rename it over the old one. A rename gives you a fresh inode, which breaks the link for that one file and leaves every other worktree’s copy untouched. This is the same trick pnpm uses on purpose with its global store; I’m just applying it after the fact.

Point it at node_modules and vendor#

For a Laravel app those two directories are the disk usage, so there’s nothing clever to do. Name the paths and let jdupes walk them:

$ jdupes -r -L ~/projects/*/node_modules ~/projects/*/vendor \
             ~/.t3/worktrees/*/*/node_modules ~/.t3/worktrees/*/*/vendor

Across ten worktrees that takes 25 seconds on this box. I run it nightly and again at the end of worktree setup, right after npm ci finishes, because the duplicate exists from the moment the install does and waiting until 04:20 wastes it for a day.

Name the paths, don’t match on directory name anywhere in the tree. Laravel also ships public/vendor and resources/views/vendor, and both of those are committed to the repo. Hardlinking a tracked file across worktrees means editing a Blade view on one branch silently edits it on every other. A top-level node_modules and vendor are gitignored build artifacts, which is what makes this safe.

The disk space saved#

The first sweep reclaimed 2833 MB. Combined with deleting the worktrees whose branches were already merged, the disk went from 98% to 57%.

If you’re provisioning a box like this from scratch, put the worktrees on a filesystem that can do reflinks and skip all of this. Real copy-on-write beats dedup-after-the-fact, because it never writes the duplicate blocks at all. I’m keeping ext4 on this box and running jdupes instead, because reformatting a root filesystem is a much bigger job than a nightly cron.