Welcome to cron.weekly issue #67 for Sunday, February 12th, 2017.
This issue was co-written by my 2 year old daughter. She requested if she could give her opinion on systemd, so here it is: gghfsi eexoàeqfkopré” ùl ca” p^ùa”ù a’”p “p^l eoircin.
But on a more serious note: it’s a big issue, which is why it took a bit longer to write. Don’t get discouraged by the length, there’s _lots _of good stuff in here. Lots of inspiration from Config Management Camp last week too.
I’d also like to welcome a new sponsor; VividCortex. See the links below for more insights.
Take your time, grab that coffee or tea, sit back and enjoy your Sunday.
News
Postmortem of the GitLab database outage of January 31
It’s no secret that GitLab experienced a database outage a few days ago. They were very open in the communication about this. This postmortem looks at what happened, how their setup is built, what recovery procedures failed and what they’re working on now to prevent a similar outage. There really aren’t that many companies that are this open about the whole thing.
POSIX has become outdated
A risky title for a risky paper: this research paper looks at how POSIX has been used/implemented in the last few decades and where the shortcomings are. I haven’t had time to fully read this, but it looks very well founded.
Container orchestration: Moving from fleet to Kubernetes
CoreOS is moving away from fleet, their current cluster manager, to Kubernetes. Yet another victory for Kubernetes that appears to be dominating the container orchestration space.
Brotli compression on Google Play Store can save up to 20% bandwidth
If you run a webserver, chances are you’ve got gzip enabled to compress the data (unless you want to prevent the BREACH attack). Google’s new compression algoritm, brotli, can save up to 20% more bandwidth/diskspace as compared to gzip. Looking forward to this becoming mainstream in nginx/apache!
Tools & Projects
Datadog: all your infrastructure, in one place
Track & alert on the health and performance of every server, container, and app in any environment, with Datadog. Sign up for a free 14-day trial. (Sponsored)
VividCortex: Database Performance Monitoring
Your database is a uniquely demanding and complex tier in your app, & needs database-specific instrumentation. See query & server performance with amazing clarity, & find performance problems that you couldn’t detect before. Start a free trial for instant value. (Sponsored)
wuzz
An interactive cli tool for HTTP inspection. Wuzz command line arguments are similar to cURL’s arguments, so it can be used to inspect/modify requests copied from the browser’s network inspector with the “copy as cURL” feature.
Beehive
Beehive is an event and agent system, which allows you to create your own agents that perform automated tasks triggered by events and filters. It is modular, flexible and really easy to extend for anyone. It’s an open source variant of IFTTT.
UPX
UPX is a free, portable, extendable, high-performance executable packer for several executable formats. It can further compress your executables, making them as small as possible.
takeover.sh
Wipe and reinstall a running Linux system via SSH, without rebooting. Takeover.sh is a script to completely take over a running Linux system remotely (to which you have SSH access), allowing you to log into an in-memory rescue environment, unmount the original root filesystem, and do anything you want, all without rebooting. Replace one distro with another without touching a physical console.
MozDef
The Mozilla Defense Platform (MozDef) seeks to automate the security incident handling process and facilitate the real-time activities of incident handlers.
backupchecker
BackupChecker is an automated backup checker.
goss
Goss is a YAML based serverspec-like tool for validating a server’s configuration. It eases the process of writing tests by allowing the user to generate tests from the current system state. Once the test suite is written they can be executed, waited-on, or served as a health endpoint.
udpcast
UDPcast is a file transfer tool that can send data simultaneously to many destinations on a LAN. This can for instance be used to install entire classrooms of PC’s at once.
HubCommander
Netflix has announced the open source release of HubCommander, a ChatOps tool for GitHub management.
Cacti 1.0
This tool has existed for so long I’m amazed there wasn’t a 1.0 release sooner: Cacti is a complete network graphing solution designed to harness the power of RRDTool’s data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box.
Wal-e
WAL-E is a program designed to perform continuous archiving of PostgreSQL WAL files and base backups.
Packer
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel.
kubeless
kubeless is a proof of concept to develop a serverless framework for Kubernetes.
Guides & Tutorials
The Unofficial MySQL 8.0 Optimizer Guide
A guide for both Dev and Ops. Quite a lot of tips and tricks for getting the most out of your MySQL 8 server!
Process reconnaissance without /proc
This post explores using inotify to do “reconnaissance” on a server without access to /proc, which occurs when your kernel is hardened with Grsecurity. A good reminder that /proc can leak a lot of data and that there are still ways around it.
Introduction to the Python programming language
This tutorial/ebook teaches you Python with a set of practical examples: it starts with the typical TODO app and moves on to introducing exceptions, code constructs etc.
Avoid ‘AllowOverride All’ in Apache to limit disk I/O access
The ‘AllowOverride All’ option in Apache enables the .htaccess feature, where you can overwrite Apache configs per directory. This post looks at the I/O impact of enabling that feature.
Replacing Queues with Nomad Dispatch
Instead of a traditional “worker queue” that handles workload asynchronously, the Nomad scheduler proposes a different method: running your jobs on-demand through Nomad Dispatch. This method has several advantages, like optimally using the available CPU/memory with as little waste as possible.
Bash Scripting The Awesome Guide
You can argue with the title, but there are some good parts in this guide: it covers loops, reading argument values, output processing, … in Bash.
NixOS Linux vs CoreOS Container Linux
A very nice summary of both OS’s with plenty of links to remote resources if you want to learn more.
5 Commands to Check Memory Usage on Linux
You know, see title – a few commands to see memory usage on a Linux server, perfect for beginners.
Monitoring and Tuning the Linux Networking Stack: Sending Data
So many details: this blog post explains how computers running the Linux kernel send packets, as well as how to monitor and tune each component of the networking stack as packets flow from user programs to network hardware.
How Unix erases things when you type a backspace while entering text
Some more details on control/escape sequences and how the kernel handles that sort of ‘special’ input.
Forgotten Linux Command Line Tricks
Some good one-liner tips here: display output as a table, repeat a command until it runs successfully, watch multiple log files at the same time, …
Using tmux properly
A very decent guide on using terminal multiplexers, in this case: tmux. I like his definition of a multiplexer, too: “A terminal multiplexer is a souped-up terminal. If you used a plain terminal for a few years and then someone said: “What features do you think we should add?”, you’d end up with a multiplexer.”
Blocking bad bots with Fail2ban
This guide covers the installation and configuration of fail2ban: this daemon monitors your log files for particular patterns, can extract the IP address and block the user for a limited amount of time based on that.
How to monitor Memcached
A good introduction to the Memcached protocol and how to interpret its data and get valuable metrics out of it.
Silly Yum Tricks: whatprovides, groups & repolist
You can use the yum package manager to search for which packages contains the file you want, you can group-install several tools at once and you can easily list which repositories are in use on your system.
Podcast
Syscast 7: Config Management Camp: Kubernetes, Sysdig & Mgmt
I recorded a new Syscast podcast earlier this week with a summary of config management camp: it covers Kubernetes, the mgmtconfig tool and sysdig.
Videos
FOSDEM 2017
Most of the FOSDEM talks have been recorded and are available online, only a week after the conference. It includes interesting ones like cgroup2: Linux’s new unified control group hierarchy.