Hi everyone! 👋
Welcome to cron.weekly issue #124.
You might’ve had a crappy week with all the Let’s Encrypt drama, I hope it didn’t cost you all too much time in follow-up.
Plenty of content for a rainy Sunday (or a sunny one, just stay inside - say “you have to work” and skip all social/family obligations).
In this episode of Syscast, I talk to Jan-Piet Mens to discuss the differences between Linux and BSD systems. We touch on the conceptual differences, distributions vs. operating systems, the packaging systems in BSD, filesystems, management via Ansible & plenty more.
This was a really fun recording since I - once again - got to ask all the stupid questions and learn a ton. I hope you’ll enjoy it as much as I did!
Listen here: 🎧 The differences between Linux and BSD »
News & general 🗞
Let’s Encrypt gave the SSL community quite the scare this week. On Tuesday, they announced they had identified a bug in their CAA DNS checks. It could have caused certificates to be issued, even if the CAA DNS records didn’t allow it.
The result was a list of potential incorrectly served certificates: a whopping 3,048,289 of them.
What do you do as a CA (Certificate Authority) when you issue wrong certificates? You revoke them! Render them useless! Burn them with fire!
And that’s what they set out to do. An update was sent out that all those certificates would be revoked just 3 days later. That gave all those 3mil+ SSL certs about 3 days to renew. Lots of sysadmins hurried to work and probably lost countless hours or days checking, verifying and renewing certificates where needed.
And with just a few hours before the deadline of revocation, Let’s Encrypt decided not to revoke them after all, because “it is in the best interest of the health of the Internet for us to not revoke those certificates by the deadline”.
Lots of panic, lots of renewals and ultimately - a non-issue. I’m willing to bet there are a lot of hours lost in the void because of this.
“If you’re using Docker, the next natural step seems to be Kubernetes, aka K8s: that’s how you run things in production, right?"
I have many mixed feelings about Kubernetes. I think it can be a great tool, if your company is ready for it. But most people that ask me if they should be running Kubernetes are small teams, less than 10 people in total, with a very static and classic web application.
Don’t overcomplicate things, focus on shipping updates and new features. The boring LAMP tech stack is fine for 90% of you.
Having said of all that … there are some interesting projects & guides on Kubernetes further down. 🙈
Lots of interesting backstory on how the “URL” came to be, looking at a brief history of DNS, the joys of Unicode/Punycode, the confusing syntax of URLs, why the double slash gets added after the protocol (
http://) and so many more interesting (but probably, useless) trivia!
This write-up provides details how Amazon has structured the metadata storage of its EBS, Elastic Block Storage. I like how the concept revolves around “minimizing blast radius".
In other words: if a piece of EBS would fail, the priority is to keep any downtime or data-loss to a minimum, impacting as few users as possible. As a result, there isn’t one large source of truth for all metadata (ie: a single database), but millions of little databases.
The current rate of public IP addresses appears to be between $20-24 per IP. This write-up covers the steps it took to sell a single /24 subnet (“Class C”, as the old folks call them).
IPv4 has become so expensive, it appears to make up about half the cost of a cheap VPS.
Here’s a bit of hopium for everyone writing open source software: a tale of success! It’s a new-ish way of writing and releasing open source packages. Those very interested in the package/project can donate and get immediate access.
Once a certain threshold of donations is achieved, the author can open-source it for everyone. If the threshold isn’t achieved, only those who paid for it get it.
Ghostcat is a serious vulnerability in Tomcat. Due to a flaw in the Tomcat AJP protocol, an attacker can read or include any files in the webapp directories of Tomcat.
“From time to time I log into the various servers under my control, and start snooping around - just looking at what’s going on, and making sure that there aren’t any unexpected things happening, and also that the stuff that should be happening, actually is happening!"
I use to do this too! Sometimes, you blindly trust your monitoring and just respond to incidents by clients. By proactively looking for trouble on your machines is a great way to prevent issues from happening!
How do you design a network with constituent hardware zooming around in space at 27,000 km/hr? I liked this post as it gave some food for thought on how to communicate with network devices that live in space.
This is good news: with the latest release, TrueNAS gained parity with FreeNAS on features like VMs and Plugins. And as of the next major version, both products will unify into a single software image and name (but still keeping FreeNAS free and open-source).
Tools & Projects 🛠
Monitor your Docker container fleet sponsored
Enhance visibility into the performance of your entire containerized environment with Datadog. Detect and monitor your containers as quickly as they are created and destroyed with the cluster agent so you can avoid resource contention and deficiency. Start monitoring Docker and 400+ other technologies today with a free Datadog trial.
ShiftLeft Inspect is static code analysis (SAST), purpose-built to insert into developer workflows without slowing them down. Inspect is 40X faster and 3X more accurate than traditional code analysis vendors. Sign-up for a free account and see for yourself.
k3sup (“ketchup”) is a light-weight utility to get from zero to kubeconfig with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get
kubectl access immediately.
This is a very simple dashboard that allows you to quickly monitor servers (ping), SSL certs, … all configured from a single, simple, JSON. If you don’t yet have a monitoring dashboard, this one seems very easy to get started with.
This project is a rewrite of the excellent Postgres Explain Visualizer (pev).
This Github page contains a series of hardened Docker containers, optimized with a good baseline of security.
This is an alternative to Google’s Chromecast. The history write-up is pretty interesting, if you care about the origins of projects.
Most online WYSIWIG (What You See Is What You Get, except in most cases it isn’t what you get - but that’s beside the point) downright suck, this is a new attempt at making an online editor that just works.
Takes an unsorted list of backup names and returns a list of backups for deletion. The backup rotation rules are given via command line args. The backup names need to contain the time and a date(1) like format specifier is required to parse those.
This pretty nifty stuff: this project allows you to build CLI applications that have a GUI layer on top of them. Hard to explain in a single paragraph, but have a read of their introductory post that has plenty of animations & videos of what the project does exactly.
Guides & Tutorials 🎓
I liked this post as it gave a good step-by-step introduction to using
fzf, the fuzzy searcher. Lots of practical commands and Bash tricks to create your own auto-completion for your favourite tools.
This promises to be a good blog-post series with all lessons learned of implementing Istio. Istio can help bring observability to a microservice architecture, providing insights into traffic flows.
In this post, we use a combination of PHP & good-ol’ native Linux tools to search the database of affected Let’s Encrypt certificates for any occurrence of our clients, so we can notify them before the revocation. It isn’t the cleanest usage of
grep, but it got the job done under stress.
This site contains a series of exercises to learn about native controls you can use to lock down your Kubernetes applications.
An easy-to-follow guide of setting up Kubernetes on your own, cheap (and as of last week, even cheaper), Raspberry Pi’s.
This post focuses on Laravel, but is actually a great introduction to Github Actions with plenty of pre-made YAML configs you could use to automate pretty much anything on Github.
A good in-depth write-up of how Github built their load balancer on top of HAProxy.