Hi everyone! 👋

Welcome to cron.weekly issue #124.

You might’ve had a crappy week with all the Let’s Encrypt drama, I hope it didn’t cost you all too much time in follow-up.

Plenty of content for a rainy Sunday (or a sunny one, just stay inside - say “you have to work” and skip all social/family obligations).

Enjoy! ☕️

🎙Podcast: The differences between Linux and BSD

In this episode of Syscast, I talk to Jan-Piet Mens to discuss the differences between Linux and BSD systems. We touch on the conceptual differences, distributions vs. operating systems, the packaging systems in BSD, filesystems, management via Ansible & plenty more.

This was a really fun recording since I - once again - got to ask all the stupid questions and learn a ton. I hope you’ll enjoy it as much as I did!

Listen here: 🎧 The differences between Linux and BSD »

News & general 🗞

Let’s Encrypt (not) revoking 3+ million certificates

Let’s Encrypt gave the SSL community quite the scare this week. On Tuesday, they announced they had identified a bug in their CAA DNS checks. It could have caused certificates to be issued, even if the CAA DNS records didn’t allow it.

The result was a list of potential incorrectly served certificates: a whopping 3,048,289 of them.

What do you do as a CA (Certificate Authority) when you issue wrong certificates? You revoke them! Render them useless! Burn them with fire!

And that’s what they set out to do. An update was sent out that all those certificates would be revoked just 3 days later. That gave all those 3mil+ SSL certs about 3 days to renew. Lots of sysadmins hurried to work and probably lost countless hours or days checking, verifying and renewing certificates where needed.

And with just a few hours before the deadline of revocation, Let’s Encrypt decided not to revoke them after all, because “it is in the best interest of the health of the Internet for us to not revoke those certificates by the deadline”.

Lots of panic, lots of renewals and ultimately - a non-issue. I’m willing to bet there are a lot of hours lost in the void because of this.

“Let’s use Kubernetes!” Now you have 8 problems

“If you’re using Docker, the next natural step seems to be Kubernetes, aka K8s: that’s how you run things in production, right?"

I have many mixed feelings about Kubernetes. I think it can be a great tool, if your company is ready for it. But most people that ask me if they should be running Kubernetes are small teams, less than 10 people in total, with a very static and classic web application.

Don’t overcomplicate things, focus on shipping updates and new features. The boring LAMP tech stack is fine for 90% of you.

Having said of all that … there are some interesting projects & guides on Kubernetes further down. 🙈

The History of the URL

Lots of interesting backstory on how the “URL” came to be, looking at a brief history of DNS, the joys of Unicode/Punycode, the confusing syntax of URLs, why the double slash gets added after the protocol (http://) and so many more interesting (but probably, useless) trivia!

Millions of tiny databases

This write-up provides details how Amazon has structured the metadata storage of its EBS, Elastic Block Storage. I like how the concept revolves around “minimizing blast radius".

In other words: if a piece of EBS would fail, the priority is to keep any downtime or data-loss to a minimum, impacting as few users as possible. As a result, there isn’t one large source of truth for all metadata (ie: a single database), but millions of little databases.

So you wanna buy a used IP address block?

The current rate of public IP addresses appears to be between $20-24 per IP. This write-up covers the steps it took to sell a single /24 subnet (“Class C”, as the old folks call them).

IPv4 has become so expensive, it appears to make up about half the cost of a cheap VPS.

Introducing Sponsorware: How A Small Open Source Package Increased My Salary

Here’s a bit of hopium for everyone writing open source software: a tale of success! It’s a new-ish way of writing and releasing open source packages. Those very interested in the package/project can donate and get immediate access.

Once a certain threshold of donations is achieved, the author can open-source it for everyone. If the threshold isn’t achieved, only those who paid for it get it.

Ghostcat vulnerability - CVE-202-938

Ghostcat is a serious vulnerability in Tomcat. Due to a flaw in the Tomcat AJP protocol, an attacker can read or include any files in the webapp directories of Tomcat.

Are you looking for trouble?

“From time to time I log into the various servers under my control, and start snooping around - just looking at what’s going on, and making sure that there aren’t any unexpected things happening, and also that the stuff that should be happening, actually is happening!"

I use to do this too! Sometimes, you blindly trust your monitoring and just respond to incidents by clients. By proactively looking for trouble on your machines is a great way to prevent issues from happening!

Network topology design at 27,000 km/hr

How do you design a network with constituent hardware zooming around in space at 27,000 km/hr? I liked this post as it gave some food for thought on how to communicate with network devices that live in space.

FreeNAS and TrueNAS are Unifying

This is good news: with the latest release, TrueNAS gained parity with FreeNAS on features like VMs and Plugins. And as of the next major version, both products will unify into a single software image and name (but still keeping FreeNAS free and open-source).

Tools & Projects 🛠

Monitor your Docker container fleet sponsored

Enhance visibility into the performance of your entire containerized environment with Datadog. Detect and monitor your containers as quickly as they are created and destroyed with the cluster agent so you can avoid resource contention and deficiency. Start monitoring Docker and 400+ other technologies today with a free Datadog trial.

ShiftLeft Inspect: Code Analysis for Dev & Ops (Fast, Accurate & Free) sponsored

ShiftLeft Inspect is static code analysis (SAST), purpose-built to insert into developer workflows without slowing them down. Inspect is 40X faster and 3X more accurate than traditional code analysis vendors. Sign-up for a free account and see for yourself.

k3sup

k3sup (“ketchup”) is a light-weight utility to get from zero to kubeconfig with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

monitoror

This is a very simple dashboard that allows you to quickly monitor servers (ping), SSL certs, … all configured from a single, simple, JSON. If you don’t yet have a monitoring dashboard, this one seems very easy to get started with.

Jitsi Meet

Jitsi Meet is an open-source (Apache) WebRTC JavaScript application that provides high quality, secure and scalable video conferences. The client runs in your browser, without installing anything on your computer.

pev2

This project is a rewrite of the excellent Postgres Explain Visualizer (pev).

hardened containers

This Github page contains a series of hardened Docker containers, optimized with a good baseline of security.

NymphCast

This is an alternative to Google’s Chromecast. The history write-up is pretty interesting, if you care about the origins of projects.

edtr

Most online WYSIWIG (What You See Is What You Get, except in most cases it isn’t what you get - but that’s beside the point) downright suck, this is a new attempt at making an online editor that just works.

prunef

Takes an unsorted list of backup names and returns a list of backups for deletion. The backup rotation rules are given via command line args. The backup names need to contain the time and a date(1) like format specifier is required to parse those.

CLUI: Building a Graphical Command Line

This pretty nifty stuff: this project allows you to build CLI applications that have a GUI layer on top of them. Hard to explain in a single paragraph, but have a read of their introductory post that has plenty of animations & videos of what the project does exactly.

Guides & Tutorials 🎓

Hacking up your own shell completion

I liked this post as it gave a good step-by-step introduction to using fzf, the fuzzy searcher. Lots of practical commands and Bash tricks to create your own auto-completion for your favourite tools.

Everything We Learned Running Istio In Production

This promises to be a good blog-post series with all lessons learned of implementing Istio. Istio can help bring observability to a microservice architecture, providing insights into traffic flows.

How we identified clients with SSL certificates affected by Let’s Encrypt mass-revocation

In this post, we use a combination of PHP & good-ol’ native Linux tools to search the database of affected Let’s Encrypt certificates for any occurrence of our clients, so we can notify them before the revocation. It isn’t the cleanest usage of split, awk and grep, but it got the job done under stress.

Want to subscribe to the cron.weekly newsletter?

I write a weekly-ish newsletter on Linux, open source & webdevelopment called cron.weekly.

It features the latest news, guides & tutorials and new open source projects. You can sign up via email below.

No spam. Just some good, practical Linux & open source content.

cron.weekly issue #124: SSL, URLs, Tomcat, monitoror, prunef & more