The differences between Linux and BSD

Hi, everyone, and welcome to another episode of Syscast, the show where we talk Linux, open source, web development, and especially for today, BSD. I don’t know anything about BSD, so I am joined by someone much more knowledgeable than me. His name is Jan-Piet Mens, or JP for short.

So, hi, JP. How are you?

Good morning, Mattias. Thank you for inviting me. I’m very well, thank you.

I’ve got a bit of a cough. I don’t think it’s corona. But anyway, we are separated by very many kilometers and cable, so you shouldn’t worry too much.

Ah, good, but then we can take breaks because I also have just one cup of coffee. We’ll see if we need it to stay awake or not. So the reason I wanted to have you on, I’ve been following you on Twitter for quite a while and have noticed that, especially over the last few months, perhaps even years, you’ve been doing more and more on the BSD end, whereas I think you originally started perhaps with a Linux background.

So I’d like to pick your brain about all of the things about BSD that I do not know. For starters, if my history of you is correct, you started at Linux and you’ve been doing more and more BSD work. Why?

Why that transition?

Well, actually, to be quite precise, I certainly did not start with Linux. I started my, let’s call it, Unix career. I’ve been going back in the calendar in order to prepare for this talk or for this podcast.

I started with my Unix career. It must have been 1986. I started actually with Unix System 5 Release 4. the AT&T side of things but simultaneously also with BSD and I can’t remember the version it must have been BSD 4.2 or 4.3 which was running on a Pyramid computer which at the time was being not produced, but sold by a company called Nixsoft Computer, a German firm, which then unfortunately folded.

And we did a lot of things Unix. On the one hand, we had the BSD systems, and on the other hand, we had the Unix System 5 systems. And 1983 must have been also approximately 1988, rather, must have been the time when FreeBSD also started.

And only then did Linux show up on the timeline 1990, I think, 1991 approximately. So my history, my Unix history dates back to basically System 5 release 4. followed by, or in parallel, Microsoft Xenix. You might remember that, or your listeners might know of it.

Yes, I said Microsoft Xenix, which was a Unix-like implementation on, what was it, 16? I can’t remember. 16 32-bit systems, I think even 16. which then later on, I believe, more or less transitioned into SCO Unix, which was also a System 5 implementation of Unix.

And only after that did I actually start working with Linux Slackware, or I can’t remember the version, Slackware 0.0 or something like that. And this must have been, as I say, somewhere around 92 approximately. So my Unix predates Linux.

Perhaps funny anecdote, but most of the things I ever wrote about on my blog, I forget about the next day because, well, I’ve written about it, so there’s no need to remember it anymore. I think you’re an active blogger. Maybe you have the same.

You just write it down and then there’s a mental capacity to just forget about it. You can always look it back up.

Well, if those words eventually helped you, I think you’ve achieved your mission. So all good. Okay, so your history goes like way back.

Maybe we can take a bit of a segue. If you look at, you mentioned Unix 5 almost 30 years ago. If you were to throw in a, say, modern Debian next to it today, can you still compare those?

Are like the basics or the fundamentals of Unix or Linux still the same? Or is it uncomparable?

Oh, yes, absolutely. Everything that is what we today would probably call user land, everything that we see on the side of the utility, so let’s say the shell, the command line tools, et cetera, that is completely recognizable, and you would have zero trouble, and I literally mean zero trouble finding your way around. A lot has changed.

So, for example, on the Unix side, we have today all the GNU utilities which have added a boatload of, well, frequently added a boatload of options or a boatload of features to the individual utilities. So, for example, if you look at a tool called grep or set, you might be missing an option or two or three or five even, but you could certainly do all your work, I think, without a problem. You would find a VI editor.

If you booted System 5 Release 4 today, you would find, or even the old BSDs 4.3, you would find a VI editor. It does not have the capabilities of a Vim editor, which you might use today, but you could wield it, you could manage it. and you would find a C compiler on these systems, and you would find all the text utilities. You would find nrof and trof, and you would, of course, not find a bash, but you would find a corn shell.

You would, I think, even find a corn shell. I can’t remember. So you would have no problem whatsoever.

That’s cool. I think that speaks to the power of the philosophy behind Unix or Linux about keeping things simple, powerful but simple, small contained binaries over large complicated binaries. I think I would struggle on the Unix 5 system today, but it’s good to know that in the end, it has evolved, but it hasn’t really changed all that much.

So that’s nice to hear.

Yes, the basics have remained the same. All sorts of changes have occurred, of course. File systems have improved, have become faster.

We have snapshotting file systems. We have today on the Linux side, we have things like ButterFS. On the FreeBSD side, we have things like ZFS, for example, or if you’re American, it’s probably called ZFS.

And so a lot has improved or a lot has matured. but the basics are also there.

That’s cool. So you were there in the very early days of Unix, Linux, and BSD. So you’ve seen them both.

I’ll talk about Linux and BSD as separate camps, so to say. You’ve seen them both grow up.

More or less. I’m hesitating because I don’t quite know how to answer your question. But yes, certainly more or less.

You said initially that I have spent very many years doing Linux. That is precise. That is correct.

So I’m what one calls a consultant. I go around. I do projects, mainly DNS, DNSSEC projects, things like Ansible.

So for a large number of years, I’ve been… I was doing projects at companies and many of these companies use Linux as their platform. So that sort of forced me more or less to concentrate on that area, maybe not so much on the other Unixes.

But yes, that’s correct.

Okay, so you were there in the early days. You’ve seen Linux and BSD. If we look at the Linux ecosystem and the BSD ecosystem today, I think it’s safe to say that Linux is vastly more popular in… perhaps the mainstream sysadmin world, BSD has become very popular for embedded devices as well.

I think most of Juniper network switches still run a version of BSD. I think my question to you is, looking back at those 30 years, can you see why Linux became more popular than BSD? Or is it not more popular?

Or am I missing something?

In my opinion, yes, Linux is much more popular. It gained… probably due, I’m not actually quite sure, but probably due to, let’s say, in quotes, marketing. It gained tremendous popularity.

It saw widespread distribution. It saw a lot of commercialization. It saw a lot of, yes, I would even call it advertisement.

I don’t necessarily mean commercial advertisement. but people who spoke about the wonders of what Linus Torvalds did. So today I would definitely say that Linux is vastly more popular than the BSDs. probably also vastly more implemented so in other words you have all the different commercial implementations you have the embedded systems you have your your lifts your your raspberry pi’s and all these things run some version of linux somehow but on the other hand of course As you rightly said, we also have a number of products that contain a BSD code starting off. I think you use a Mac.

In a Mac, there’s BSD code in your Juniper routers. If you watch Netflix, Netflix is powered by free BSD. I believe they have…

They use FreeBSD and they also develop on FreeBSD because they are able to massively tune the network stack in order to provide the bandwidth that they actually need to distribute all these masses of video. So there are a large number of manufacturers of products that contain, let’s say, free BSD or one of the BSDs, not last because of the licensing. The licensing is on the BSD side very simple.

You typically have the two or three clause BSD license, which typically allows you to do anything you want with the code.

Okay. So if I can circle back to the thing I wondered before, you have that 30 years of experience on Linux and you’ve been more and more active on the BSD end. Why that shift?

Why going back towards BSD?

Well, it’s maybe a little bit of a religious answer to the question. The I mentioned earlier that I have done all sorts of projects at customer sites, and these are typically customers which use any of the, I’ll say, large Linux distributions. That’ll be Red Hat slash CentOS, that’ll be Debian slash Ubuntu, or it might be SUSE Linux Enterprise.

And to be honest, this is maybe or very possibly coupled with an age thing. But I have been getting more and more, shall I say, upset about the constantly changing interfaces. By that I mean, for example, if I need to set up a network in, uh, or if I needed to set up a network on just an interface in red hat, it used to be, uh, in sysconfig somewhere, there was a network script and in Debian, there was an interfaces file and so on.

And, um, all these things, all these interfaces are constantly changing. Now this might be for the good of mankind. I’m not able to say so.

Um, but it has, uh, It has not helped me. And I suddenly decided that I wanted to change that for myself. So I wanted to go back to, I wanted to do something different.

And that different, that is one of the BSDs, say OpenBSD or FreeBSD, which are far more traditional in their changes. They do not constantly attempt to revamp the world. I’m not sure if I’ve answered your question, but that is one thing.

And another thing which has made me revisit OpenBSD, revisit FreeBSD, for example, is I have decided that since I occasionally write a little bit of code in whatever language it may be or occasionally write the odd system, I wanted to be able to do stuff in a portable way. So I decided earlier this year that whenever I wrote a line of code, I would make quite sure that that would run on at the very least two, I think three, I said, different operating system platforms. So that would be, for example, one of the BSDs, maybe a Linux distribution.

And yeah, so those are mainly the reasons why I’ve done that.

That is indeed a very nice clickbait title. It is to a certain extent, it might even be approaching the truth. To tell a very short story, I basically think a lot of the things that SystemD has provided or is providing, are good.

I know a number of people who are going to say, JP, how much did you drink this morning? But a number of things that systemd implements are good. For example, systemd unspawn is very interesting.

Nobody uses it or hardly anybody uses it, but it’s very interesting. Lightweight containers on the systemd or the timers in systemd instead of using cron, for example, or the unit files, the fact that we can sort of just drop in a unit file and have that activated somehow. Those are things which I personally think are very good.

They are very useful and it was the right thing to do. The reason that I’m not specifically very friendly or reasons why I’m not specifically very friendly with SystemD, A, I am a strong believer that SystemD was implemented too quickly by the large Linux distributions without it having experience enough, let’s say, quality control or history. So it was very, very quickly, if my recollection is right, very quickly rolled out onto basically all the Linux distributions, or at least very many of them.

And there are certain things that I would like to say intensely dislike, like the journal, which basically would be a good idea, but the way it’s been implemented, the fact that I have to search around looking for logs, et cetera, those are things that I dislike about systemd. Yes, I will admit that.

I think I’m in full agreement there. I like the very same parts that you mentioned. I think the power of a unit file is really great.

We’ve been able to dump most of our hacky bash scripts that we just call sysv init, but they were just bash scripts. I mean, everybody writes them in their own way. There’s hardly any consistency.

I think systemd for the most part has improved things. I also curse at my interface names. They are, quote, predictable.

But really, if you call eth0 and eth1, that’s predictable, not some random MAC address-derived naming scheme. yes that is yeah so i i think i curse and praise at systemd for the very same things that you do um and i think if you want to be or if you want to use a pure um linux slash unix system i think your transition to bsd makes sense especially as you said in the beginning if most of the tools are essentially similar um then perhaps such a transition from linux to bsd Isn’t rocket science either? Or is that harder than I imagine it to be?

No, definitely not. As I already said earlier, if you had to go back to a System 5 Release 4 system today, we probably wouldn’t necessarily want it because there are all sorts of things, all sorts of tools missing that you might want today. But no, this transition is really very easy.

I mean, it starts if you install a free BSD system, for example, if you install an open BSD system, you might say, oh, my goodness, it’s a text installer. How am I going to survive that? Well, who please needs a graphical installer to set up an operating system?

The amount of money that has been invested by these people is just gigantic. Installing an operating system is something that you typically do either in an automated fashion if you’re an enterprise or that you do just by hitting a few keys. Nobody needs, to my mind anyway, nobody needs a graphical installer.

So the installers might, on OpenBSD, on FreeBSD, for example, they might appear to be slightly primitive, but they make sense. They work. They are reliable.

And once you’re up, then you’ve got a Unix system. You’ve got… basically you’ve got something which is in a way quite different to Linux. Linux is a distribution.

That is something that, to my mind, very many people don’t really understand. I mean, your listeners probably do, but anybody who starts off doesn’t necessarily understand that. This term distribution means…

The kernel is developed by an entity. The utilities are developed by a second entity. And then you get a third entity.

These are called Red Hat or Canonical or Debian or whatever they’re all called, who take these individual bits and pieces, assemble them, and bring them out as what they call a distribution, which is fine. I mean, it works or typically works. It’s no problem.

But if you install an OpenBSD or a FreeBSD or a NetBSD, by the way, there are also too many of those, what you get is an operating system. So if you install a FreeBSD, for example, you have a full operating system that comes out of one entity, out of one hand. And that operating system is not only the operating system.

You have the full source code to the operating system. You have… When you’ve done your installation, you’ve got a compiler, you’ve got an editor, you’ve got all the utilities that you need, or most of them anyway, which belong to the base system.

It’s a complete thing. It’s not individual bits and pieces that have been put together. Of course, you can then add packages you can add external programs obviously but a base OpenBSD or base FreeBSD is something that is fully functional and has been developed in concert is has been developed together so if the makers of FreeBSD decide to change something they can do so and they can be sure that the next deployment of this operating system will work.

It will be one thing.

Yes, it’s possible that that could be said about, for example, Canonical or maybe even Red Hat. I’m probably not familiar enough with the way they work, but they get… I think they get the kernel from upstream and then they patch it around or they modify it to suit their needs.

Isn’t that right?

Ah, yeah. So I think where you make that distinction, I think it makes sense to me now. Canonical takes a lot of independent pieces from different upstreams, different developers, different projects, bundles them up and ships it as Ubuntu, whereas OpenBSD or FreeBSD…

They write their own kernel. They write their user-land-based packages. They have control over that ecosystem that they will ship as a new version.

That is correct. Is that a correct assumption? That is correct.

Then I think I will make a comparison that will get me a lot of hate. In a sense, OpenBSD is more like Windows than Linux.

Okay. It’s also been nice talking to you, JP. I think you will lose about half your followers now.

No, they were all gone. Okay. You mentioned BSD and you mentioned the different flavors already, free BSD, open BSD, there’s net BSD.

How do those differ? What’s their origin and what do they target?

Well, the origin, I think all the origins were at BSD 4.3. So we’re talking now approximately 1990, 1991. And then there was a company or an organization.

I seem to recall the man was called Bill Jolitz, Bill Jolitz, something like that, who created something called 386 BSD, which was BSD running on an Intel chip, on an Intel 386. And out of that then came a free BSD. I think today we would probably call it a fork.

So for some reason… And I’m sorry, I do not recall the reasons. These people weren’t very happy with each other, and then FreeBSD was created from that.

Out of that also came an operating system called NetBSD, also in 1992, 1993 approximately. And one of the people who worked with NetBSD, or maybe even founded this operating system, was a man called Theo de Raad. And he then split off and basically said, I don’t want to work with you guys or I’m not happy with the way this project is going.

And he again forked this NetBSD and turned that into an operator system, which today we know as OpenBSD. OpenBSD, this is the project and these are the people exceedingly clever people who are also who have also been responsible for example for OpenSSH for OpenSMTPD for a number of spin-offs which I think originated with OpenBSD so it’s an operating system which is deemed or which is termed as highly secure it’s quite small it has a number of I could imagine you might say limitations. I would call them a number of, let’s say, features.

So, for example, OpenBSD, the developers of OpenBSD refuse to implement Bluetooth. support in their operating system which already for a number of people would mean okay they can’t use it because they can’t connect their latest whiz bang bluetooth thing now there’s a reason of course why openbsc people don’t want to implement bluetooth and that is that bluetooth is inherently insecure so they don’t even want to touch it and so the openbsc is a very strict very rigorous team of different people people basically like you and i but very clever programmers who develop and who have also managed to create exquisite, and that I really mean the way I say it, exquisite documentation. I have never seen better manual pages for a Unix system than on OpenBSD. It’s really, really, really exquisite.

I was demonstrating or showing the other day Our user suggested a patch, basically a three- or four-word patch to a man page, and then one of the developers chimed in and said, we should maybe do that differently and phrase it this way and phrase it that way. So there are people who speak up about documentation or change documentation who spend quite a lot of time working on the documentation. The man pages are really quite exquisite.

By the way, also something which is slightly lacking, I think, on the Linux side. So, yeah, that’s basically the history. So 4.3 BSD, then 386 BSD, and then split off into these three projects over a certain time, over a few years.

I think it was four or five years in total. Today we have basically FreeBSD, NetBSD, OpenBSD, but then there are others. So there’s something called DragonflyBSD, and there are a number of additional spinoffs.

I think most of them spun off from FreeBSD.

I really do not know the numbers, but I would guess that it would be FreeBSD. OpenBSD is a little bit special. I very much like it because it’s, yes, it’s small and it’s… beautifully written.

It’s lovely code. It’s very reliable. I very much like it.

But I think FreePSD would probably have the highest numbers, certainly in terms of let’s say, installations on maybe laptops or servers. I do not know about embedded. Also, I would actually guess FreeBSD, but I’m not familiar enough with the numbers.

I think FreeBSD would probably win that war. Well, it’s not a war, but those statistics.

Okay, my gut feeling would have said that OpenBSD was more popular. I have no idea why, but looking at the tech news, tech articles, it could also be because, as you mentioned, OpenBSD is perhaps a bit more radical in its ideas that it… gets written about a bit more because, well, if it’s more extreme about certain ideologies, it usually gets covered more as well. But I see OpenBSD pop up more than FreeBSD.

Or at least that’s my gut feeling.

Very hard to say. It might be that you perceive it that way. It might also be the case.

As I say, I’m not familiar with numbers. My gut feeling would be free BSD. Also in terms of topics that I’ve heard, for example, at two of the BSD conferences, I had the pleasure to be invited to BSD Can, which is a BSD conference in Canada, in Ottawa.

When was it? Last year. And I’ll be going again this year.

I’ve been accepted to give a talk about DNS servers. And I was in Lillehammer at the EuroBSDCon in November last year. And there…

If you look at all the talks, I would say it’s mainly FreeBSD or, yeah, a focus on FreeBSD.

It depends. If possible or if feasible, if I think it’s sufficient for whatever I want to do, I would probably tend to open BSD. There are a number of things in FreeBSD which are…

Yeah, killer features. Number one is ZFS, the file system, with its snapshots, with its boot environments. That’s just, I never use the A word, but that is really awesome.

And number two is certainly jails, the lightweight containers that FreeBSD has to offer. where on a laptop with 8 or 16 gigabytes, you can create 200, 300, 400 jails and run individual applications in there, sort of containerized. Those are features that FreeBSD has, which OpenBSD does not and most likely never will because of a number of reasons. But yeah, most likely never will.

There’s no reason to implement them. So I would say, Typically, to answer your question, my go-to would typically be FreeBSD.

Okay, because it gives you a better base to build any application on. Correct. Or host any application on.

Okay, cool. I think I have a good idea on the history of those BSD versions, I’ll call them. If you work on them day-to-day, so you’re SSHing to… free BSD a net BSD and an open BSD would you be able to tell the difference straight away or would you only notice that by a lack of certain binaries certain features can you distinguish them at first sight I would I would hazard a guess that it would take you quite a while to notice the difference

And I can actually corroborate that by saying the following. I gave a training two weeks ago, and the trainees used – they connected via SSH to a bunch of servers. And after – it was a four-day training.

These were system admins, so system admins more or less familiar with Linux. And at the end of the four days, as I was packing up, I asked these people because I wanted to know. I wanted a little bit of their feedback because this whole environment I had completely newly built, I wanted to know.

And I asked them, was everything okay? Did you have any trouble? Did you notice anything specific?

And they all said, no, these are really, really cool Linux systems. It was free BSD jails. So they didn’t notice. if you, Mattias, logged in, you would very possibly notice very quickly.

So, for example, if on that system you’d done a DF disk-free, you would have immediately seen, oh, that’s a strange-looking device name that doesn’t appear to be very Linux-y. Or if you did a PS, you might notice a difference. And, of course, there are certain utilities that taste a little bit different.

There are certain command line utilities which you might use, as I mentioned earlier already, which may, in the GNU world, in other words, in the Linux world, have an additional option or two, where FreeBSD or OpenBSD would say, don’t understand that option. But I think it would likely take quite a while for you to notice.

I think you’re right. I can imagine nowadays everybody should be using config management and automation and we should not be installing packages manually on servers. But let’s assume that you still do.

The moment you want to install a package, I’m guessing you would notice because the package manager between a Linux and a BSD device would be vastly different.

Well, with all due respect, the package manager between your Linux distribution and my Linux distribution is completely different as well. Correct. Yeah.

So while I agree that a lot of us or a lot of large environments do config management or configuration management, of course, there are probably tens of thousands, maybe even millions of people who’ve never heard of it and who wouldn’t want to because they administer one or two machines, maybe their own laptop. And if you are running, what is your preferred Linux distribution? Do you have one?

I am shifting. It used to be CentOS, but I’m shifting towards Ubuntu. Okay.

So all the better, actually, that you say that, because… You used to type yum install, and now you type apt-get install or apt-install. So between these two worlds, you have a difference.

And not only do you have a different utility name, which, by the way, is – well, that’s a long story. Not only do you have a different utility name, but you also have completely different semantics. So, for example, in the YUM world, or it’s not called YUM anymore, it’s called DNF now because let’s have a new package manager, it’s a new release.

In the YUM world, you have in the DNF world, in other words, in the Red Hat or CentOS world, when you do an install, by default, you have a repository refresh, which happens in the background or in the foreground. On the Debian side, that does not occur. The repository refresh only occurs when you do an apt update.

So you have also on… on the individual or over the individual Linux distributions, different utilities, different syntaxes, different semantics for doing things. And interestingly, many people say, oh, but BSD is different. And I say, yes, sure, it’s different.

But all your Linux is also different. By the way… going back to the beginning of our conversation, one of the reasons that drove me back to BSD. So, yes, to come back now to your question.

Sure, if you logged on to my free BSD machine, then you would ask me, what’s the package manager called? And I would say it’s called PKG. So you would then type in package, PKG, install, package name.

And then the package is installed. It’s downloaded from a repository and installed just like it would happen on your… CentOS on your Red Hat or on your Debian machine.

I think the naming in FreeBSD already makes more sense because you’re installing a package, so you name the package manager package. I was hoping you’d notice. Yes.

I have no idea why someone would call it DNF or what even is YUM? It doesn’t make any sense.

YUM is, if my memory serves me, Yellow Dog Update Manager. Yellow Dog was a… Oh, but now memory fails me.

Wasn’t that a Taiwanese Red Hat implementation? Something like that?

The documentation, by the way, is really excellent. You have a very good free BSD handbook, which, for example, goes through all these things. But yes, it’s a package upgrade or a package update, which will do exactly what you expect it to do.

And you can also update all packages. For example, on OpenBSD, there’s also a package… i think it’s package package no package info minus you which will upgrade all installed all packages that you’ve installed so you know all these all these systems have utilities which do what you would do on a linux system typically do all those bsd systems have the same package infrastructure package manager or is there a difference between yum and apt as well in the bsd ecosystem unfortunately i think i must say there is also a difference here so the the package infrastructure on openbsd differs from that of freebsd for example the the utility name is different the the the What are called the ports and the packages are slightly different. So yes, each of these worlds has their own difference.

They are separate operating systems. They are not flavors. They are separate, distinct operating systems.

So in the Linux world, we’d probably say they are distinct distributions. In the Unix world, I say they are distinct operating systems, and therefore… They display differences.

Okay, so it’s hard to put them side by side and compare them. As you mentioned, they’ve been around since 1990-ish. So they’ve had a small 30 years to evolve in different directions.

It would probably make sense that today… they are vastly different operating systems. They just happen to share the BSD naming and perhaps a very old kernel at the root of it. Okay.

Since you’re talking about packaging, I think I saw you mention that you’ve written packages before for BSD. Is that correct?

Yes, that is correct. I became interested earlier this year, two or three months ago. No, that’s not true.

This year hasn’t had two or three months yet. I think it was end of last year. I became interested, respectively, I had the necessity to… provide a package or two.

And I wanted to do so on OpenBSD and on FreeBSD. I’ve done a bit of packaging before in my earlier life. I think it was Red Hat 4.

No, not RHEL 4, Red Hat 4. And creating packages is about the worst thing I’ve ever done in my life. And I wanted to sort of experience how is it in the BSD worlds.

So, yes, I submitted two or three packages. One was a port of a utility which I wanted to use, and one was an own piece of code. And, yes, writing packages for these two systems, FreeBSD and OpenBSD, is…

Well, I won’t say trivial because packaging never is trivial, but it’s basically following quite good documentation, quite adequate documentation. And then when you submit a port, what is called a port, in other words, a new package for them, When you submit a port, you basically submit a piece of make file, which will, in the infrastructure of the individual operating system, which will then cause that, what you’ve provided, to actually turn into a package. It was… interesting not something i would like to do every day but it was interesting not particularly bad i don’t recall it as being let’s say better or worse than any other packaging system i’ve seen what i liked a lot is it’s um The tooling is basically a whole bunch of make files and things built around makes or around components that a Unix system already brings with it.

Okay. Stupid question from my end. Do the packaging systems on BSD, do you ship…

Pre-built binaries or because there’s a compiler available in every system, you ship source code and it gets compiled in those make files. What gets shipped?

You can do both. The FreeBSD, for example, has what is called a ports collection and OpenBSD likewise. And this ports collection is… very large directory tree which has some somewhere like i think 20 000 packages at the moment so 20 000 directories split over different categories so you have system category network category miscellaneous mail whatever and um you can, as you rightly say, because we have a compiler on these systems, because we have the whole infrastructure that is necessary to actually build something, which is not necessarily the case on a Linux system, where typically nowadays compiler is not installed by default, which is very sad, but anyway, because we have the whole build infrastructure, you can actually descend into one of these directories, say make the original tar file, is downloaded, then the software is unpacked and it is made, it is built.

And you are able to, typically anyway, configure individual flavors of the package in a make file. So I’m just inventing something. You might have a program XYZ, which has a flavor XYZ and it has a flavor XYZ with TLS.

Yeah, you could decide when building which flavor you would like and these ports are converted centrally converted into packages downloadable binary packages and whoever runs that infrastructure decides on the flavor so it might be that you could install via package where you get xyz installed But if you wanted the special TLS flavor or the special ABC flavor, you would need to build the package yourself or build the port yourself.

So if you want utmost flexibility, you would build from ports. However… What you typically do is you build a package yourself from the port and then install that package locally.

Okay. So it’s like on your own machine, building your own RPM files and then installing them. And there is also, for example, FreeBSD has a package builder called Poudrière, which runs in jail.

So it runs sort of… containerized, let’s say compartmentalized. And with that, you can, for example, for your organization, for your company, set up a package builder, which automatically retrieves new ports, builds packages, and makes them available for distribution via the package system, via the PKG utility within your own organization or over your own server infrastructure. So the tooling is…

I would say the tooling is very good. And in particular, the tooling is simple. It’s not rocket science.

It’s not something you have to study for days on end to get to know it.

Oh, it will work and you will not run into problems at all, I would say. It will work for the simple reason that, for example, you can… on FreeBSD, decide whether you want to install Apache, whether you want to install Nginx, or whatever other piece of code you want to install. There are, I think, momentarily somewhere around 20, 25, maybe even 30,000 packages from which you can install binaries, or binary packages which you can install.

And with the exception of maybe the odd path, which is going to be different, I think you wouldn’t notice. For example, if you install an Apache on CentOS, you get slash etc or slash etc slash httpd. If you do the same on Debian, you would get slash etc slash Apache 24, or Apache 2.

Apache 2, indeed. and if you install an apache on freebsd you would get slash user local etc i think it’s then httpd so the the configuration paths are different but other than that i think you wouldn’t you wouldn’t notice a difference

Okay, well, I think you mentioned a good segue here. So if you’re used to running Linux, you know that most of your configuration files will probably be somewhere in slash etc. What do you say?

Is it slash etc or slash etc?

I think there are about 17,000 different ways of saying it. I personally say slash etc. But there are people, I had a poll about that about a year ago.

There are people who say ETC. There are people who say et cetera, which to me is too long. And there are, I think it’s mainly Americans, and I like it, but I’ve not been able to adopt it.

I don’t know why, who say Etsy, which I like because it’s very short.

I’m camp slash etc, so I’ll go with that. If you install software and you’re looking for configurations on Linux, you would probably start looking at slash etc. You mentioned on FreeBSD, it would be slash user local slash etc.

What path differences stand out for you between Linux and BSD?

The formula is basically if the component of your configuration file is provided by the operating system. then it’s in slash etc. If it’s provided by something you install, so some package or some port which you explicitly install, then it would be typically located in slash user slash local slash etc. So that’s a differentiation.

This differentiation is done so that the operating system and the ports slash packages can…

Yes, that is correct. In terms of free BSD, what I think we would typically do is basically not care. If you’re about to do an upgrade, for example, you could create a boot environment, and this boot environment will have a snapshot of your operating system.

And if the upgrade fails for whichever reason or renders your system unusable, you can boot into a previous environment and then you basically get, it’s like a snapshot of your whole operating system of which you can carry multiple and you can boot into these individual boot environments to come back to a well-known state. So, yeah, what you say is correct, it’s possible. It’s, I think, something that’s not done very frequently, but it’s certainly possible.

Which makes it a lot more clean than, say, I think Ubuntu is trying to change that with its snap packages where everything tries to be contained in different directories. But by default, a Linux system is a mix and match of custom-based applications, some source compiled applications, some operating configs. They’re all all over the place, basically.

BSD is better structured.

I think it’s better structured. I personally agree. I might be biased, of course, but I personally agree.

And what I particularly like about, for example, FreeBSD in that respect is you have slash etcrc.conf, which is your startup configuration file. So in there, you basically define which services you want to enable. Do you want send mail?

Do you want this? Do you want… what’s your interface called? How do you obtain your IP address?

All that is defined in this basically shell script type file of just individual settings. And you have the same structure under slash user slash local slash etc. So you have exactly the same thing there.

If you provided a piece of software via port or via a package or just manually you could drop a startup program startup script into user local etc rc.d i think it is and um freebsd will automatically pick it up when it for example when it reports away when you use the service program to start to stop or enable a particular service so we have on the one hand slash etc and on the other hand slash user local etc but with the with very similar or actually identical tooling which is used on both ends so that makes it quite effective quite quite easy to to master i think

Oh, no, no, no, no, no, no, no, no. I didn’t do that. Oh, I don’t.

I’m not good. I’m never going to write a book again. I wrote a book about DNS sec.

I’m sorry. I wrote a book about DNS servers, about open source DNS servers. but not about Ansible. I have read several books about Ansible, and on the back of the Ansible up and running book in the second edition, my name is on there, basically saying he read the book.

Stamp of approval that you’ve read it. Okay, let me rephrase. It’s safe to say that you know your way around Ansible.

I think this is where you say, yes, I know my way around Ansible.

I’m nodding. You can’t see me nodding, but it’s safe to say that I’ve heard about Ansible. Yes, correct.

I started working on Ansible way back when, I think it was 2012, when Michael DeHaan originally created, or very shortly after he originally created this tooling. And I did a number of contributions to Ansible. I wrote a few of the modules.

I created the documentation system, so the system by which even today the Ansible modules, for example, document themselves that was originally I created that. So I know my way, a little bit of my way around Ansible, yes, and I do trainings that is correct.

Okay. With your Ansible experience, do you see a big difference between using Ansible to manage a Linux versus a BSD system? Or because of the abstraction layers, it feels the same?

How would you categorize managing Linux or BSD with Ansible?

That’s a very nice question. I like that question because I think that more or less – or the answer could more or less sum up a lot of what we’ve been talking about now. I very much like that question.

There are differences, of course, and these differences are mainly due to the – underlying target operating system so let us suppose we have an ansible controller which is the the system on which ansible is installed and i’m going to now target on one hand let’s say a freebsd system and on the other hand let’s say a santos or debian system there are initially some differences of course One of the differences, for example, is that if we want to have privilege escalation on the target systems, we would need to install sudo, for example, or do as on FreeBSD because FreeBSD does not come with these utilities. So that is something we would need to install if we wanted to use that or if we required to use that. On the other hand…

For example, FreeBSD would require an installation of Python because FreeBSD does not by default, I think, come with a version of Python. Let’s assume we’ve done that. Let’s assume, for example, during installation, we’ve ensured that we have those prerequisites. then on the Ansible controller, we have relatively little that we need to take care of.

But yes, of course, there are differences. If you write what is called a playbook to deploy software or to deploy configuration onto these systems, you will have to account for certain differences. So for example, on your Linux machine, you will, as we just spoke earlier, you will create a file called slash etc slash httpd slash whatever.

And on the FreeBSD side, that might be called slash user slash local slash etc slash httpd slash whatever. So we have to account for these differences. That is the reason why Ansible For example, it has its fact variables, which tell us which flavor operating system are we on, specific paths, IP addresses, all sorts of technical information about the systems that we are deploying onto.

So if you spend a little bit of time, not too much, if you spend a little bit of time thinking about this and preparing, then the end result is basically that you, yes, you can deploy onto either flavor. Now, there are inherent differences. So, for example, this happened, I think it was mid last year.

I had a playbook which needed to run make, the make utility on the target system. And it worked flawlessly on Linux. and it failed on FreeBSD. Now, the reason it failed on FreeBSD is FreeBSD by default does not come with GNU Make, whereas Linux comes with GNU Make.

These are two completely different tools. So you have the BSD Make on the one hand, and you have the GNU Make on the other hand. GNU Make has a very large number of additional capabilities and options.

The reason the Ansible module failed is the module… the Ansible component, the Ansible module, was invoking make with dash dash file make file name. But the BSD make does not recognize this option dash dash file. The BSD make only knows an option dash F.

I was able to fix this, of course, very quickly locally, and I then provided one of my trivial patches, which was then applied. to change the module in the Ansible distribution to only use “-f”, which would then do exactly the same thing on both sides. So there are differences. Some of these differences we can very easily work around.

Others require maybe additional installation. So you have, for example, the unarchive module in Ansible land, which is able to extract a zip or a tar or compressed tar file. Now, this unarchive module, the unarchive Ansible module, has been written, apparently, by somebody who was only familiar with tar on Linux.

And tar on Linux, that is GNU tar. which has a boatload of additional options compared to the original TARs or to the FreeBSD TAR. So the unarchived module, in order to execute that on a BSD machine, requires the gtar package. Now, this is documented.

It’s documented in the module. So in other words, your playbook would need to account here again for the FreeBSD machine and would need to additionally install the gtar package. So these are the kind of things that we… that we have to realize, that we have to know.

Here again, to my mind, it’s important that this is possible. It is important that we can use Ansible to distribute to basically any Unix-like operating system. And yes, we can even do Windows now.

And these operating systems are different. And I think what is most important for me is that people who write Ansible modules should maybe be more aware, if they aren’t already, but made aware of the fact that there are differences in the Unixes and the target Unixes, and try and accommodate for these differences, if at all possible. So, back to your question.

Is it possible or is it very difficult to run Ansible against different Unix versions, against different Linux versions? The answer typically is no, it is not difficult. There’s a few things that you need to look out for, a few things that need configuring, for example, the path to the Python interpreter.

But other than that, we can run playbooks, and I do this a lot. I teach this also in trainings since about two and a half years. I’ve been doing trainings again.

I’ve been teaching. I’m hugely… I’m hugely liking this again.

And so these are things that I very explicitly point out in my trainings, that one has to attempt to be portable.

So managing a BSD through Ansible is not much more difficult than, say, managing CentOS versus Ubuntu. have paths to keep in mind. Your binary names might sometimes differ. Your package names might sometimes differ.

But if you’ve already been managing your configs, whether that’s Puppet or Ansible, which targets Red Hat or Ubuntu, targeting a BSD next to it. If you’ve already had the pleasure of finding the differences between Ubuntu and Red Hat systems, finding the differences between a BSD system shouldn’t be rocket science anymore.

Yes, I would say for a lot of what you do, or no, let me rephrase that, for a lot of what I think you do, There are certainly reasons to do so. FreeBSD, for example, or OpenBSD on a server is something that you can do immediately. FreeBSD on a laptop, on a desktop is something that you can also do immediately.

I must say, though, I’m not specifically familiar with that. For me, a laptop is… I live in a shell, so a laptop is… if I have Tmux and a bunch of shells, normally speaking anyway.

But yes, these are definitely systems that you can run. FreeBSD, for example, OpenBSD, very reliable. I’m not trying to insinuate that Linux is less reliable.

I’m just saying FreeBSDs are very, very, very reliable. We have… You even have now Amazon EC2 machines running FreeBSD, so you can install FreeBSD there.

I’m not familiar with that, though. What I have done is digital ocean droplets with FreeBSD, and the result is, yeah, it’s a BSD machine. So these are things that…

I mean, if you think that companies such as Netflix, which earn their money basically off being able to transport video and a boatload of video.

use FreeBSD, then it can’t be all bad. No, and if you imagine that pretty much any Juniper router runs BSD and they can have uptimes of five years or more, okay, they should have been rebooted for patches, but that’s beside the point. I think it’s safe to say that FreeBSD, OpenBSD, they can be rock solid and very stable.

Absolutely.

If you imagine FreeNAS, for example, is one of the… major open-source NAS network-attached storage systems. FreeNAS runs FreeBSD. So, I mean, yeah, there’s a large number of products which build on the shoulders of these operating systems.

So, yes, definitely have a look. I think you wouldn’t necessarily regret it on the contrary. I’m not trying to convert you.

This is not a religious drill here. But yes, definitely have a look. I can recommend to anybody to have a look.

Start off with a, if you like, with something virtual. Start off with a virtual box or with your VMware Fusion or whatever you have and install OpenBSD or install FreeBSD. It’s a matter of literally minutes to do this installation.

Don’t get put off by, as I mentioned earlier, don’t get put off by a text installer on the contrary. I think it’s actually a good thing. because it means that they’ve invested time and money elsewhere. And, yeah, have a look around.

I think it’s safe to say that after logging on to a BSD system, you’ll be…

rather hard-pressed to very quickly determine that it’s not Linux. I think you’re right that the text-based installer doesn’t matter. I think for most use cases, nobody ever sees an installer.

If you go to DigitalOcean, well, you never see it. It just gives you an IP address and you SSH into it and it works. I also don’t really care about Linux on the desktop.

I know there’s been like a decade of effort of making it happen. And I know it works for some people, but I just don’t really care about Linux on the desktop. I think it is an excellent server operating system.

And I only look at it as a server operating system. So I think you’re right. I’ll give FreeBSD a shot.

I mostly run web servers with PHP. So that’s got to be one of the most popular use cases for any operating system running PHP somewhere. So I’ll bet I will be able to find the guide to take me by the hand. and install through ports or through package all the packages that I need to get some Apache and PHP going.

This will be a fun experiment. I’ll give that a shot. And if I’m ever stuck, I will poke you for advice.

Absolutely. Please, please do. I mean, there are very, very many people who are much, much more knowledgeable than I am.

But if I can help you experience one of the BSCs, then I would very gladly do so. It’s… It’s a nice experience.

Some people might think, oh, this is a bit strange. It all feels a bit old and maybe even specific packages only exist in a specific slightly older version. Yeah, okay, fine.

Then that’s the way it is. But of course… as you and I can very easily do. We can, of course, grab the newest version or we can build from a port, whatever.

These systems are, yeah, they’re solid. And even if the virtual software versions might be a bit older here and there, I have decided that I can live with that. And not only that I can, but also that I would like to live with it.

Well, I come from CentOS slash Red Hat, Red Hat Enterprise Linux, sorry. So I’m used to having very old and stable software. It is never cutting edge or never up to date.

It’s just secure. So maybe I won’t notice a difference with BSD.

JP, thank you very much for the chat. I think I’ve learned a lot about BSD. Maybe you’ve converted me halfway and I’ll see if I can get there for the other half.

So thank you for the chat. It has been a real pleasure for me. If the listeners would like to keep in touch with your online movements or your professional movements, how can they follow you on Twitter, a blog, LinkedIn?

How can they reach you?

Well, not LinkedIn. I mean, I am there, but that’s not a platform for me. If you want to read about what I do, the best way is probably my blog.

That’s jpmens, mens as in ladies, .net. And if you do not shock easily, then my Twitter account, that’s at jpmens, same name, is probably the way to follow along with my rants and with my love and hate for all sorts of different topics.

Ah, that might be it. I’ll add the links to your Twitter and homepage to the show notes as well. So if anyone is looking for the direct links, go into your podcast player and see if you can find the notes.

Click through. Definitely follow JP. I actually liked following you because of all the DNS news that you share.

I don’t know how you find the time to read all of the PowerDNS release notes and everything and just pick out the useful tidbits and throw those out. Keep doing that because I love it. It makes sure that I don’t have to read through every release note.

Yeah. For obvious reasons, I also like DNS. So it’s nice to see you care about that as well and see you share those updates.

Okay, JP, I think this is a good time to wrap up. We’ll talk to you later. Bye-bye.

Bye-bye, Mattias. Thank you very much.