Automating the Unknown

Mattias Geniar, Wednesday, December 3, 2014

While Config Management isn't new as a concept, it is new for a lot of sysadmins. Many have started using it in the last year or two and have invested in workflows, modules, re-usability, ... to keep their systems up-and-running. Yet, at the same time, there's a disturbing trend I see emerging of automating something you don't fully understand.

I see this in 2 variants: trying to automate a service someone doesn't fully understand, and blindly copying modules and code, implementing it on systems and not understanding what goes on under the hood.

Automation = knowledge + insight + a goal

Whenever you write a Puppet module or a Chef Cookbook, you're essentially writing in code what you think is the best way of implementing, managing and securing a service or role. You're pouring your years of experience, your knowledge on the subject, your insights into the technology and your long-term goal into that module. Your module / cookbook / ... should reflect your view of how that service or role should work.

That's only possible when you truly master said technology. You can automate a piece of technology you do not understand, but you shouldn't. It creates technical debt from day one and provides the foundation for more bad practices. It's a downward spiral that's hard to get out of, because chances are you're using that piece of code in production.

It can happen for a variety of reasons, deadlines and management pressure would be the most common. You make the choice of getting the job done and you make it work. Short-term, the automation pays of: you've got something up-and-running. Long-term, you're stuck with a badly implemented technology and a dependency on the code you've just written, with no time to undo your quick hacks. Classical technical debt.

The reality is, automation takes time. A lot of time, if you want it done right. Not the automation in itself, but understanding the technical details, the shortcomings, the workarounds and the best practices of a particular technology. And you can not write a module or cookbook for any piece of technology, if you do not master it.

Using unknown automation

So if you lack time to master technology, surely you can just grab a module or cookbook from the forge or the Supermarket and be done with it, right? Let other smart people write the modules with their years of experience and knowledge. You just implement it and benefit from their work.

This work, up until a certain point. If you just take an already written module and use it, do you understand what that module is managing? Do you know the internals and how to debug it? How will you react when your service is slow to respond or offline? Do you know where to look and where to apply the fix?

Implementing modules that others have written, for technology you do not master, is like following a guide on the internet with step-by-step commands. It'll get the job done, but you won't know why certain steps are taken. You won't know why things are done the way they're done. And when you really need it -- when your service crashes -- you won't know where to look. Because the internals are a black box.

It takes commitment

And I don't mean a git commit. Building a config management infrastructure, writing and managing modules and implementing them in a correct way takes serious commitment. It takes time, effort, knowledge and motivation. It means caring about code quality.

While it's a gut feeling, I dare you say you shouldn't try to automate a service if you haven't used for a few years "manually". If you've not had to debug it yet, have experienced the shortcomings of a typical "first time deployment" and not had your lessons learned, it's better not to automate it in the first place. It won't bring you any benefits (do you need repeatability of a bad setup?) and will only slow things down (how do I migrate my legacy module to this new and improved way of managing I just came up with?).

Truth be told, config management isn't easy.

Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!


Filip Van Tittelboom Thursday, December 4, 2014 at 12:28 - Reply

Totally agree! Except maybe your conclusion, but I know where it comes from.

I also have been calling this blind use of modules as ‘copy-paste administration’.

A lot of expertise, frustration, bug-fixes and tricks go into daily ops of services which don’t get necessarily translated in readily available external modules.

Another fact is that a lot of (puppet in my case) modules out there (practically most of them imho) are poorly written and bloated. There are some exceptions.

Granted, you have to watch out by not re-inventing the wheel all the time. Mastering a technology is key to determine this.

Ryan Baker Friday, April 10, 2015 at 00:31 - Reply

Agree with Filip, the conclusion is missing out on the opportunity that automating the unknown affords. You can iterate from unknown to known if you have automation.. but if you just manually replicate the same unknown.. well good luck fixing those mistakes.

I think however where you can take your insights is, declaring victory after automating the unknown is clearly wrong. Using unknown automation in any situations other than a) extreme trust in the source or b) absolute desperation or c) learning (somewhat the same thing as b), is a bad bad idea. But creating it can be fine.

Maybe where this is coming from is an experience where you, or someone you worked with, was trying to learn how to automate at the same time they were trying to become an expert in what they were automating. That could be pretty difficult.

Leave a Reply

Your email address will not be published. Required fields are marked *

Inbound links