I forgot how to manage a server

Mattias Geniar, Wednesday, May 1, 2019

Something embarrassing happened to me the other day. I was playing around with a new server on Digital Ocean and it occurred to me: I had no idea how to manage it.

This is slightly awkward because I've been a sysadmin for over 10yrs, the largest part of my professional career.

The spoils of configuration management

The thing is, I've been writing and using config management for the last 6-ish years. I've shared many blogposts regarding Puppet, some of its design patterns, even pretended to hold all the knowledge by sharing my lessons learned after 3yr of using Puppet.

And now I've come to the point where I no longer know how to install, configure or run software without Puppet.

My config management does this for me. Whether it's Puppet, Ansible, Chef, ... all of the boring parts of being a sysadmin have been hidden behind management tools. Yet here I am, trying to quickly configure a personal server, without my company-managed config management to aid me.

Boy did I feel useless.

I had to Google the correct SSH config syntax to allow root logins, but only via public keys. I had to Google for iptables rule syntax and using ufw to manage them. I forgot where I had to place the configs for supervisor for running jobs, let alone how to write the config.

I know these configs in our tools. In our abstractions. In our automation. But I forgot what it's like in Linux itself.

A pitfall to remember

I previously blogged about 2 pitfalls I had already known about: trying to automate a service you don't fully understand or blindly trusting the automation of someone else, not understanding what it's doing under the hood.

I should add a third one to my pitfall list: I'm forgetting the basic and core tools used to manage a Linux server.

Is this a bad thing though? I'm not sure yet. Maybe the lower level knowledge isn't that valuable, as long as we have our automation standby to take care of this? It frees us from having to think about too many things and allows us to focus on the more important aspects of being a sysadmin.

But it sure felt strange Googling things I had to Google almost a decade ago.

Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!


T Wednesday, May 1, 2019 at 13:49 - Reply

Isnt ufw just an abstraction of iptables? How far down the rabbit hole do you go?

exhausted_sysadmin Wednesday, May 1, 2019 at 13:53 - Reply

I felt the exact same thing and first installed Gentoo on a vm to refresh my memory, then switched to nixos on my laptop to force myself to try something new. I can’t recommend enough.

Hakan Bayindir Wednesday, May 1, 2019 at 14:36 - Reply

You could have also used the manual pages installed on the server. It has all the examples you need, at least for sshd and other well-aged services.

Heil Honkler Wednesday, May 1, 2019 at 14:53 - Reply

Real men use iptables. ufw is Ubuntu filth.

philbert Wednesday, May 1, 2019 at 15:01 - Reply

Great post! I’m where you were 10 years ago. Just now getting started with Ansible and I often wondered if this happened to folks.

jonty Wednesday, May 1, 2019 at 16:12 - Reply

Your experience with config management tools is just the latest layer at the top of a pile. Twenty years ago we were all making the transition to package managers (rpm, dpkg) from building utilities from their source code (tar -x; ./configure; make; make install) . And thirty years ago we expected to create our own build routines (vi Makefile; make; pray and curse).

Don’t worry that you are “forgetting the basic and core tools”. There was a time when those tools didn’t exist and now you are standing on the shoulders of the giants who built them for you. When a taller giant appears it is natural to climb a little higher so you become more productive.

xte Wednesday, May 1, 2019 at 16:26 - Reply

You fall in modern (ancient) commercial trap, I’ts hard to get out unfortunately…
GNU/Linux evolution is a mess, like commercial products, so use it in comfort is a mess
so you (we) normally look for other ways.

*BSD was a niche, despite niche, options, OpenSolaris a hope, but for a reason or another
they can’t be used anywhere. The modern mania of developing crappy thing worse than
’90-style software, running for worse on someone else computer’s complete the pictures.

My personal recipe for now is do my best to be “single OS” with NixOS (but still, I’m on
NixOS but have also other OSes to work with) and try to look for OpenBSD or FreeBSD

It’s time, for our society, to start thinking a big re-write of our tech rediscovering ancient
technologies like LispM, Plan9, even plain historic unix model. But that’s only an utopia
these days.

Bharrison Wednesday, May 1, 2019 at 16:36 - Reply

I have such an irrational fear of a scenario where my life depends on an awareness of current command syntax that setting up a bash alias requires a massive internal struggle and sleep-on-it resolution before being implemented.

This article makes me uncomfortable.

Scott Robinson Wednesday, May 1, 2019 at 17:41 - Reply

Happens to folks who use a bunch of aliased command too. They forget the underlaying commands and syntax for ease of use.

Rob Wednesday, May 1, 2019 at 19:04 - Reply

There are stacks and stacks of abstractions in tech; as new abstractions get built we both individually and collectively forget how the lower level stacks works. Managing stack frames and building data structures used to be a common part of programming; now we have higher-level languages and easily importable libraries.

The same goes for system administration. We built automation layers so that we don’t have to remember all this stuff – the need for extensive Googling and years of experience to internalize knowledge is a bug, not a feature, of the software. I wouldn’t be sad that you’ve forgotten the old knowledge since your new knowledge is that much more powerful.

StarPeace Wednesday, May 1, 2019 at 21:03 - Reply

Does it means that you user Windows or OSX as your daily driver on your laptop or workstation?

Stephen S. Musoke Monday, May 6, 2019 at 10:12 - Reply

I would say that is a good thing, and it is okay to have to go back to the basics once in a while.

The exciting thing is usually that you find your own information on the Internet on how to resolve the issues you are facing.

Vahid Mahdiun Thursday, May 16, 2019 at 19:17 - Reply

As matter of a fact, it is nature of human being to forget old information , as you know, the more you repeat tasks the more connections will be generated between your neurons so to speak. bottom line is, i believe everyone has this issue of forgetting essential work-related stuffs. as a rule of thumb, repetition is the ultimate solution to fix this stupid bug in your brain. i use both configuration management tools and manual command for my servers and projects. i presume there must be a combination of both so your brain muscles stay active.

orfie Thursday, July 4, 2019 at 23:03 - Reply

That happens way too often I guess.. My remedy is trying to devote some sessions once a week to setup a new software package from scratch, just for the sakes of revisiting some fundamental knowledge.

Leave a Reply

Your email address will not be published. Required fields are marked *

Inbound links