Matt Holt, creator of Caddy webserver

You can subscribe via RSS , on iTunes , Spotify or any other Podcast player.

Image of Mattias Geniar

Shownotes for episode 1, published Wednesday, 9 May 2016

(Note: audio quality isn’t super great, I’ll get that fixed for episode #2)

Exciting! This is the very first episode of the SysCast podcast!

I’m very happy to have Matt Holt as my first guest, who created the Caddy webserver .

We talk about the Caddy project, the Go language and community, the unique benefits of Caddy and its Caddyfile, dealing with Pull Requests and so much more.

A couple of lessons learned on my part:

  • stereo recording is useless, everything should be mono (I now merged stereo to mono, but it came at a loss of audio quality)
  • I need to tweak my own recording settings, that’s obvious (it’s a good thing Matt’s voice is clearer than mine)

Shownotes#

Transcript

WhisperX large-v3 + pyannote diarization, lightly edited.

Mattias Geniar

Hello and welcome to the very first episode of Syscast. My name is Mattias Geniar and today I am very happy to be joined by Matt Holt. Matt, how are you?

Matt Holt

Good. Glad to be here today.

Mattias Geniar

I’m glad that you joined us. Apologies again for the very random timing on my part. Let’s hope that never happens again.

That’s all right. Well, some people may know you, some may not. You’re mostly known for a very specific project that you did, the Caddy web server.

It’s interesting in a lot of ways. So I’m pretty curious to talk to you about how it came to be, why you’re still doing it, what motivated you. But first, could you introduce yourself?

How did you get started into programming and perhaps go in specific?

Matt Holt

Sure. Yeah, so I grew up in Iowa in the United States, outside of town, kind of in a rural area. And I didn’t live with siblings, or I don’t remember living with my older siblings, at least, when I was younger.

So it was just me and lots of farmland. So I got into programming, got a few books. And so I’ve been programming for a little over 16 years, which…

It isn’t very long compared to a lot of the great programmers I know, of course. But it’s been fun. I started using Go at work about three, maybe four years ago now.

And it’s been fantastic. I’ve been able to use Go for most of the projects that I need to do, except for web pages, which I still use JavaScript for in the browser. But Go has been a fantastic language and able to serve most of my needs, at least.

Mattias Geniar

Well, if you’ve been doing this for a couple of years, then I think you’re one of the early adopters of Go. No, it’s not that old.

Mattias Geniar

Huh?

Matt Holt

It’s not. I think version 1 came out in 2009 or 2011. But I think some of the veteran, like the senior level Go programmers are probably been doing it for about six years now, six or seven.

Mattias Geniar

They probably invented the language in the first place. Impressive, impressive, really. So you develop Caddy, the web server.

How did that start? What made you decide that we should have a new web server on top of all the other ones?

Matt Holt

So I was still in my undergrad at university doing computer science, and I was in a really difficult semester. So I kind of needed a project to keep me kind of sane on the side. I couldn’t be totally enveloped by school.

So I started working on a web server. It’s not that I decided that we… generally needed a new web server, but I needed a new web server. I needed something to match my workflow to do what I wanted to do and work in a way that I wanted to develop websites.

I just found it a little tedious to be setting up a web server with a centralized config every time I wanted to work on a new website, especially dealing with compilation issues and Just complex syntax and configuration nuances and stuff. I just wanted a very nimble kind of workflow. And so that’s why I started writing Caddy.

I wanted something with a little higher level control than the web servers that are popular right now.

Mattias Geniar

That I can understand, especially if you’re just wanting to build a simple website and you think, okay, I’ll deploy Nginx. It usually takes a couple of hours. Then you have a working version and perhaps then you can get started.

So I think from that point of view, Caddy makes a lot of sense because that has simplicity written all over it, which is I think then one of the reasons you decided to make it, just to make things simpler, to get to production faster and simpler.

Matt Holt

Yeah, I believe, although I have no research to back this up, I believe that the majority of websites don’t have complicated needs. If they’re well-designed, your needs aren’t complex. And I bet that Caddy can serve a lot of websites well.

Mattias Geniar

I can definitely understand that. So Caddy, what are some of the unique features that you managed to implement in Caddy that were missing from the other ones?

Matt Holt

You know, Caddy doesn’t have so many unique features as it does a unique workflow. There are a few things that stand out, though. Something, again, I know my own itch that I wanted to scratch was just being able to write markdown documents for simple websites that I don’t have to use HTML for.

So Caddy converts markdown to HTML on the fly during a request. It’s fairly fast. at doing that. So that’s really nice.

And then its security features are, of course, fairly unique. Being able to generate certificates automatically and serve your site over HTTPS by default is a very unique thing. Caddy is the only general-purpose web server, I suppose, to do that.

It can even issue certificates during a TLS handshake. I call that on-demand TLS.

Mattias Geniar

um well that part is absolutely crazy because if if if you’re a listener and you don’t really know what that means it’s like you configure a website and the very first time that you as a visitor browse to that website it’ll fetch its certificate or it will make a certificate signing request send that to let’s encrypt it’ll get back a certificate and before the user notices it or at least i think there’s probably a noticeable delay but well negligible in the grand scheme of things but but After that, it’s a valid SSL connection. You have a valid screen certificate. All good.

Matt Holt

Yeah, and I mean, that depends on the speed of the certificate authority as well. But it’s been interesting to kind of see that being used. And then, of course, it does all the usual…

OCSP related things and even Caddy recently now added TLS session ticket key rotation and security feature to preserve perfect forward secrecy and Caddy’s the only web server to do that out of the box as well.

Mattias Geniar

Could you describe in layman terms what that means? If I browse to a certain website, where does that come in?

Matt Holt

So when you make a TLS connection, your browser and the server negotiate some secrets that they share so that they can verify the integrity of the connection and encrypt the communication. And if someone is recording these encrypted transmissions, so there’s the idea of forward secrecy. Now, by the way, I’m not a security expert.

I’m not a cryptographer. Most of this just comes from my basic security class from my undergraduate college course, but what I understand is that perfect forward secrecy, when properly implemented, can protect previously recorded… TLS sessions from being decrypted if the key is later stolen.

And by rotating the keys, it’s kind of like flushing a cache, I think. And so as I understand it, when you rotate the keys, it just limits the window of an exploit if someone is able to obtain a secret. that window is drastically narrowed down. Whereas if you don’t rotate the keys at all, it could last for the entire lifetime of the server and someone has a lot more time to break in.

Mattias Geniar

Okay, so it adds a random element to existing TLS sessions as well. Is that correct? Something like that, yeah.

That I can see can improve security indeed. So Caddy is easy to use, secure by default. That all sounds really good.

Well, I’m a sysadmin by nature. I think judging by the website that you created, you’re aiming mostly at developers to have a very low barrier to entry to get to use Caddy. Me as a sysadmin, why should I prefer Caddy over something like an Nginx or an Apache?

Matt Holt

That’s a great question. So I am not, as a disclaimer, I am not actually a sysadmin. I don’t declare myself to have expertise or skills in that field.

So for the developer, of course, Caddy is appealing because it’s kind of a nimble, agile, as in portable… that kind of adapts to your workflow very easily. As a sysadmin, I suppose a lot of what you do is involved with setting up systems, right? Like Linux, presumably, and doing deployments.

Mattias Geniar

Exactly.

Matt Holt

Yeah, so if that’s your main job, then Caddy is really easy to deploy. It’s an entirely self-contained Go binary, and this is just a virtue of being a Go program. There’s nothing magic here that I’ve really done.

But Caddy is statically compiled, so you can run it on any machine that Go compiles to, even if that machine doesn’t have libc installed. And you can still get pretty much the full benefit of Caddy and all the Go standard library for the most part. So it’s a single binary.

It’s really easy to ship and deploy. And it still lets you do, again, everything, most of the things, you know, like that 75% to 85% of what you would need from Nginx or Apache most of the time, I should say.

Mattias Geniar

That I can indeed also imagine. The fact that it’s a single binary, that everything’s statically compiled makes, from a deployment point of view, everything so much easier. Especially now we’re on the verge of a Chrome update that is about to break HTTP2 for a lot of web servers because it’s going to switch the method in which it negotiates the protocol.

And the simple, stupid change that it’s going to do is going to require an OpenSSL update, something that is usually tied very deeply into the system because of the dynamic linking that’s going on. Making an upgrade to something like OpenSSL is very tricky because you could upgrade it and the upgrade can go fine, but every program linked against it or using its API could suddenly break. having everything compiled in a single binary that you as a developer control from beginning to output is I really like it I can imagine you as a developer as well just to have something that once you build it and it works in your system it’s going to work on mine as well

Matt Holt

Oh, yeah, it’s fantastic. Let me tell you, I actually, so I haven’t done, there’s a lot of hype about like Docker and containerization and, and that kind of thing. And I haven’t really been caught up in that hype.

Because again, most of my work for my needs has been in Go and And shipping Go binaries is, I mean, you don’t need a container, for example. You don’t need to manage dependencies in production like you would with a dynamically linked web server or any other dynamically linked software. And another kind of neat thing, too, is that Go programs are fairly resilient to a lot of the common programming errors that you find in C programs.

Buffer overflows and bad memory accessing and things like that don’t really happen in Go. It’s a little bit more memory safe in that sense.

Mattias Geniar

That’s perfect for deployments as well, indeed. I think one of the other main contributors to I think caddy’s popularity and its future growth that I’m predicting is that you bundle the entire configuration into a single config file that you call the caddy file. It’s comparable to something like a vagrant file or a make file in the sense that it’s a single file that contains your entire configuration for the web server.

And you just have to type caddy in the right directory, the directory in which you’ve stored the caddy file. And the web server starts with the configuration that you or the developer or whoever decided that the web server should have. Was that something that’s a deliberate choice from the start?

Something to make it as simple as possible?

Matt Holt

Yeah, it was. I’m glad you caught that. It’s inspired by the vagrant way of doing things and the make way of doing things.

CD into the folder, run the command, it finds the file in there to use as configuration, and then it just works. one of the, one of the things that I, I advocate against when people, you know, talk about distributing caddy or packaging it up for certain distributions is like having like a central place for a caddy file. That’s like all filled out with all these commented examples and, and, And I kind of discourage that because one of the things I’ve always had a hard time with, not being a sysadmin, remember, is how to find… So if I install Nginx, great, where do I go to find its configuration?

I would rather just cd to my site’s folder and just have its configuration live by the site. So when I ship my site around, the configuration goes with it. To me, it makes more sense that way.

So, yeah, I feel like that’s a very important part of Caddy’s kind of agility and its ease of use.

Mattias Geniar

yeah i’ve been thinking about that particular use case a lot in the last few days it’s a perfect way for a developer to actually ship an entire web server configuration in its most simple form so instead of having to include an entire nginx config that nginx could then include you’re shipping an entire web server configuration for caddy that doesn’t need to be included it’s just the entire configuration as is which if we’re looking at say deployment for a new php website if you have particular rewrite rules or you want to do that markdown to html conversion you could write a caddy file put that in your git repository and tell your sysadmin hey this is my web server config and it can just run caddy and everything works i find that rather magical as a sysadmin in the positive sense

Matt Holt

Oh, good. Yeah, and I should say, too, that Caddy definitely tries to make your job as a sysadmin easier, although it can’t do your job for you. So the day I launched Caddy, and I showed people on Hacker News, and people were visiting it, and the website, of course, was powered by Caddy, this very early prototype version, and the site went down.

And that’s a little embarrassing, right? It is, it is. But it wasn’t because Caddy was incapable or was doing something wrong.

It wasn’t leaking resources even, but it ran out of file descriptors. It hit the system user limit. So I forgot to raise the U limit for file descriptors.

Now, if you run Caddy on a production site and you forget to raise that limit to a high enough value for most sites, it’ll actually warn you. A little message there now.

Mattias Geniar

That’s a save default, yeah. I think another benefit that I’m not sure how long it’s been in caddy, I only just recently noticed it, is that you have the ability to use environment variables within the caddy file. So if you’re a developer and you want to test on your local machine a web server on port 8080, because that’s convenient to you, but the web server is either going to run directly on port 80 or going to be proxied on a variable port.

We sysadmins still have the ability to choose which port you’re going to bind on by just setting an environment variable. I think that’s best of both worlds where you can still, as the developer, completely organize and configure the web server the way you like, but the sysadmin has the ability to run multiple versions of caddy on multiple ports. and perhaps puts an Nginx or a load balancing proxy in front of it. So that’s a really perfect use case to me.

Matt Holt

Yeah, that’s great. And Caddy, we want it to be environment aware. It has to use its environment.

It has to live and work inside its environment. What we don’t want to do, we have to draw this careful line between… adapting to its environment and being scripted. I don’t want to get to the point, I don’t want to cross this line where you have to script the caddy file.

And if you’re going to be scripting or writing code or logic and having memory or something, at that point we want to… just recommend that people use a programming language like Go or Node or something to write a web server. But again, we’re going for like the 75% of use cases that are out there.

Mattias Geniar

Which brings me, you mentioned a very good topic, memory usage, which relates to CPU usage and just resources in general. How does Caddy handle this? It’s multi-threaded by default, I think, right?

Matt Holt

It is, yeah. So we use Go’s concurrency model, which is Go routines. These are lightweight threads, so a new Go routine does not necessarily spawn a new system thread.

The Go scheduler can in some ways be smarter than the system’s thread scheduler because it understands Go code, it understands input-output needs and timers and these other kinds of things. And so whatever time the system gives the Go program on the CPU clock, Go is going to utilize that to the maximum efficiency. And so its scheduler is very good.

They’ve done excellent engineering with that. And so it’s all self-contained in a single process as well, which is nice. You don’t have to be keeping track of all these processes and demonizing it even.

So… And maybe that’s something we should talk about is running caddy in production.

Mattias Geniar

I’d love to, if you have any best practices for that, I’m all ears.

Matt Holt

Well, I, again, as I’m not a sysadmin exactly, I can’t, I don’t, I’ll tell you, I’ll tell you what I do. I’ll tell you how I run the caddy website and my own websites is I actually just, I run caddy usually with no hub, no hub caddy, and then just run it in the background, put an ampersand at the end. Um, Now, there’s disadvantages to that, I understand, but it is a very good way to run it.

It’s never brought my site down. That won’t necessarily survive a server restart, but you could use an upstart script or something like even a cron job can run a script when the system restarts that just runs that command. It’s really easy.

There’s a lot of discussion and a lot of problems. It’s really easy to do this wrong where you use a process manager, even system D, unless you really know what you’re doing on the sysadmin level.

Mattias Geniar

caddy is not as hard to run as people think i think um i think your examples made that clear it’s a very simple config file run caddy and you’re good to go i think what you mentioned is indeed the challenge of keeping it running no hub is great but gives you perhaps a bit less control of the process but even a job server like supervisor or god or any of the alternatives would be perfectly suited just to run caddy

Matt Holt

It can be, yeah. Just know what you’re doing. So, but Caddy has, I mean, as far as, I have not received a report that Caddy has crashed hardcore in production and needs to be restarted.

It doesn’t really need to be supervised in that sense. Not saying that nothing will ever go wrong or that there won’t be a bug someday, but it’s very reliable as is.

Mattias Geniar

If you’re running the Caddy web server directly onto your server, does that mean it’s running directly as root to bind on the lower ports?

Matt Holt

You can do it that way. I don’t recommend it just because of good practices is to not run as root. You can use the Linux utility setcap and give caddy the capability to bind to low ports.

That’s typically what I recommend. Some other people get around this by running caddy in a container and then using IP tables to forward ports. You can do that however you’d like.

Mattias Geniar

Indeed, there are plenty of ways to do it, one better than the other, or perhaps more how it’s always been done, what you’re comfortable with, what you’re comfortable managing. I think being a sysadmin is partly trusting what you know and how you’re going to do it, and a very small part of leaping into the dark, especially in production, it’s always a bit tricky to introduce… either new software or new ways of running things um especially uh if it’s there’s no alternative um so if i’m currently running my own websites on a combination of apache and nginx if i wanted to try caddy since i’m running my systems on a single server there’s no really there’s no alternative to me but to shut it down and start it with caddy and pray everything works So it’s a bit, well, potentially invasive if you make a mistake to replace your web server. With a tool like Caddy, I could actually just make a Caddy file for each of my projects, set up a proxy the layer above it, so say on my Nginx, and try out Caddy without, well, possibly destroying all of my websites or projects.

I think that makes Caddy a very clean project to get started with or to experiment with. I really like that approach.

Matt Holt

And you could, I mean, just thinking on the spot here, depending on your site and your needs, you could even just run Caddy, set it up to run your sites on port 81 or some other low port that’s not in use and just try it yourself that way on a different port. And then if it seems to be working, then you can switch it out. For simple sites, something like that might work.

Mattias Geniar

Indeed, indeed. What are some of the crazy use cases you’ve seen Caddy be used in, if there are any?

Matt Holt

You know, it’s funny. People don’t tell me a lot when they’re using it. Sometimes people are very excited.

I see Caddy used a lot for personal sites and small business sites, and that’s fantastic. That’s what it is really good at, and it can do bigger stuff too. I don’t know.

People, I think, hesitate to share they’re using new technology sometimes. I don’t know. So I don’t have an idea of use.

It’s just what people tell me. Caddy doesn’t phone home. But a couple things do come to mind.

Recently, Caddy was used. So Caddy is used by a lot of Go programmers as well. And a few weeks ago, me and another Go developer, Brian Kettleson, We saw a tweet by a Go developer in Cuba, and he was asking for help to access the Go website and blog from Cuba.

It was blocked because it’s on Google’s infrastructure.

Mattias Geniar

That’s a very serious block if everything on Google is being blocked.

Matt Holt

Yeah, apparently. Because the Go website is totally politically neutral. Anyway, Brian and I felt like, well…

There’s no reason we can’t. We’re in the United States. We can do him a favor.

So we just changed a caddy file. We added literally three lines, and one of those was a closing curly brace to the file. So 30 seconds later, we had this reverse proxy set up to the Go website and the Go blog for the developers in Cuba to be able to access Go documentation.

Mattias Geniar

That’s amazing with just three lines of code. If I wanted to do the same thing in either Apache or Nginx, first I’d be Googling like crazy, comparing different versions with other different versions and then getting syntax errors and then trying to figure out which kind of proxy I need to do. And I do this for a living.

I set up reverse proxies daily. I do not know how to do that by heart. It’s always a matter of going back to documentation or Googling.

If you can do that in three lines, well, actually two, if you exclude the curly brace, I think that’s really amazing for a web server.

Matt Holt

Yeah, and it was actually an encrypted reverse proxy as well. It was on its own subdomain over HTTPS, and that took 30 seconds because Caddy issued the certificate automatically. So any man-in-the-middle attacks would not be able to succeed because it’s encrypted via TLS, and so you can’t know exactly the content that’s going there if they were watching that.

Mattias Geniar

I applaud you, sir. Very nice.

Matt Holt

Well, thank you. It wasn’t totally, you know, it wasn’t me alone.

Mattias Geniar

No, but still, Caddy is your baby. So the fact that it can do what it does today is perhaps not only in part of you, but you’ve supervised it. You’ve had to make decisions either before or against the current state of Caddy.

So where Caddy is today is entirely up to you.

Matt Holt

It’s been a great community effort too. I have seen Caddy, like people have tried to use it as like a certificate manager because of its TLS features and how it can obtain and store certificates on disk and keep them renewed. I don’t recommend using Caddy as like a certificate manager.

Just for that, use it as a web server. Yes, the certificates are placed on disk, and you can use those in other servers that you need, like a mail server that uses StartTLS. Go ahead, as long as you know what you’re doing.

But don’t rely on Caddy just for the certificate management stuff. Use it as a web server.

Mattias Geniar

Well, one of the more popular certificate management tools written in Go is, I think, Lego. Was that in part forked from Caddy or is just entirely independent?

Matt Holt

So Ligo was started by Sebastian Erhard. He’s a developer in Austria. He’s a computer security student there.

So that started because I was looking for someone to help implement the Acme specification in Go for use in Caddy. And Sebastian volunteered and took charge of the project, and it’s an incredible project. The code is pretty clean, and it’s…

But yeah, it was originally just for Caddy, but of course it can be used generally by any Go programs.

Mattias Geniar

That just goes to show what kind of additional tools can be spawned by just, well, you have a need, you try to fix it. Before you know it, the world has a better Acme slash Let’s Encrypt client. So cheers for everyone.

Matt Holt

It’s the easy, well, I’m a little biased. It’s the easiest to use. He’s done a fantastic work on it.

And it’s also one of the oldest or more mature Acme implementations out there. Probably just second to the official client.

Mattias Geniar

anyway so well the official client has a very big downside that it’s rather how to put this delicately dynamic just launching the tool could run all kinds of things in the background like suddenly installing a gcc compiler additional python libraries without you asking for it which is it works the client does its thing but it does a lot of things in the background that i as a sysadmin wish it didn’t do So Lego being a single Go binary is, again, a very big benefit. If you want to deploy that, it’s a lot easier than having to run either ancient Python versions or just a compiler onto your system. The official client has some downsides.

Matt Holt

Yeah, and I think recently they actually don’t have technically an official client anymore. They’ve moved that over to, I think, the EFF. It’s kind of incubating that now.

But we did actually reference, Sebastian did reference the official client when making Lego because, of course, they know it best. But the nice thing about Go, like you said, is very easy to deploy. It runs instantly, no need for dependencies.

Mattias Geniar

Yeah, indeed. So going back to Caddy, Caddy obviously supports HTTP2, the newest version of our protocol. One of the perhaps debated is a heavy word, but one of the more lesser implemented features is server-side push.

Could you explain what that is and where Caddy stands in implementing it?

Matt Holt

Yeah, so HTTP2 server push is a mechanism by which the web server can preemptively or predictively send resources to the browser before the browser asks for it. So if the server sends an HTML page, it can also realize somehow that the client is going to need this image and this style sheet, this JavaScript, and just send those down the wire right away. And we’ve seen that this… or decreases, I should say, the page load time.

So it does have a positive performance impact. The tricky part is the implementation is not really defined in the specification, which is good, but it’s up to the web server to use heuristics or some sort of configuration that… so that it can know upon a certain request for a resource if it needs to push anything else. So there’s a lot of questions to answer there.

There are implementations out there that are fairly new. They work. They’re new.

Caddy will get one eventually, but we’re not yet.

Mattias Geniar

Do you have any idea what kind of most of the web servers that currently offer server-side push usually do that by receiving some kind of additional header from the developer’s application and then translating that header into the HTTP server-side push protocol? Most of them today now use the link header. So it’s very easy to set as a PHP or a Ruby developer.

Just add an additional header to your application to tell what additional resources could be pushed back to this client. Do you have any idea how Caddy would implement that? Would you follow the same routines?

Do you have other or better ideas?

Matt Holt

I’ve thought about it a little bit. I’m actually not an HTTP2 expert. I haven’t implemented the spec myself.

I’ve looked through it. So I like the link idea. I think that’s fine.

The caddy mentality, of course, is to take the approach that uses the least amount of configuration and requires the least amount of maintenance. So I will, of course, be closely watching to see how the Go standard library implements server push, if it ever does. I think there’s at least one open issue about it.

But I understand they have a lot to do already. But if it’s going to take a while or whatever, we may look at our own implementation. I don’t know exactly how it’s going to work, to be honest.

I thought it would be kind of cool if Caddy is serving an HTML page, if it could just kind of quickly parse the… at least the head portion of the HTML, and then just kind of read the links that it needs and just push those automatically. But again, I don’t know what implications that has or how good that would be.

Mattias Geniar

Well, when the spec was mostly theoretical and not very practical, I first read that part as if the web server could do that. But then, of course, HTTP comes out, Nginx, Apache, everyone starts to integrate it. But the lack of server-side push is one of the biggest noticeable ones out there, because I think server-side push was one of the features that got a lot of attention.

Perhaps one of the things that got developers excited about the new protocol, a more efficient way to push your assets, your CSS, your JavaScript into the browser. But it’s then, on the other side, one of the most lacking features in most other browsers. I think the implementation that you described i’m having the web server parse the head or perhaps even the dom entirely sounds really really interesting and at the same time so complicated for you as a developer to make and to make a good accurate decision on which asset needs to be pushed to begin with or is on the same domain or is served on the same web server there’s so many tricky edges but the idea sounds so enticing

Matt Holt

It is enticing. And, uh, and you know, server push is actually like, although there is a positive performance impact, in my opinion, not the most important part of HTTP two. Um, I think more importantly is that you have the connection, uh, the multiplexing, uh, the requests on the same connection, uh, and, uh, and the security requirements of HTTP2 as well.

So server push will be nice to have, but in a way, we kind of already do it when we… when we… Oh, what was the word? There are hacks for HTTP 1.1, and some of those hacks kind of emulate server push anyway, so most people’s existing sites aren’t going to see a whole lot of benefit.

Mattias Geniar

but some i agree i think um some bigger websites already did some kind of benchmarking with server push and i think the results at this point are very mixed for for some it’s heaven and everyone should be using it for others it’s uh it works but it doesn’t really give you that much of a performance increase caching is another issue but i don’t even want to think about that right now there is there’s a new spec in town about clients being able to tell the web server that they already have a filing cache so they don’t receive the push the downside is more work for every web server to implement that spec on top of http2

Matt Holt

Yeah, HTTP2 is a complex protocol.

Mattias Geniar

Yeah, indeed. I think the biggest benefit you already mentioned, the fact that everything’s encrypted now, that headers can be compressed, that we’re multiplexing. I can attest from the server point of view, if there are web servers that are running at 1,000 or 2,000 requests per second, enabling HTTP2, even though it’s not available for all the browsers yet, has a noticeable decrease in CPU time on the web server.

So for busy web servers, I think it’s definitely the way to go. It’s stable enough for now just to enable it everywhere.

Matt Holt

And it should save some file descriptors too.

Mattias Geniar

Indeed. And a lot less weird-ass TCP session states or upper bound port limits that we’re reaching. It doesn’t really solve them, but it eliminates a bit of the problems or it makes them harder to hit in production.

So that’s a good thing.

Matt Holt

And it should cut them down by a third to a sixth.

Mattias Geniar

Yeah, indeed, indeed. You can still hit the limits, but it’ll be harder to hit the limits. So what are your thoughts on HTTP2 in general?

Are you happy with the current state? You mentioned that it’s not really your area of expertise.

Matt Holt

Yeah, it’s good. HTTP2 is fantastic, I should say. And the Go implementation is quite good, considering its age.

It will, of course, become more mature later. HTTP 1.1 is, what, 20 years old now? So HTTP2, let’s give it some time.

I think, let’s see, if I remember correctly, there was one obvious omission, which was protocol upgrades. So if you wanted to, over HTTP2, switch to WebSockets, I don’t think you can do that with HTTP2. I might be mistaken.

I haven’t looked at that in the last almost year. And then we were also having some issues with… and I think this might be a Go-specific issue, not necessarily an HTTP2 spec issue, but we’re having some issues with browsers keeping the HTTP2 connection alive until the browser is closed. Like a very extreme interpretation of persisting the connection to the point where it’s interfering with… with certain caddy process like restarts so if you reload the config file caddy can do that gracefully but anyway there’s some weird behavior going on with http2 and long-lived connections and restarting so like edge cases yeah but you should be too is great it’s new so i’m guessing uh give it a couple of years or just a couple of months uh we should especially now that more and more people are actually implementing its server side as well we should see vast improvements coming soon

Mattias Geniar

right yeah switching back perhaps to the more of the the human side of what caddy is um you’ve invested a lot of time into caddy is that a full-time job for you now is it still a hobby project how does that go

Matt Holt

It feels like a full-time job sometimes. It has kept me busy. I love it.

It’s a lot of fun. But no, Caddy doesn’t pay the bills, of course. I have contract work for that.

And I’ll be starting grad school in the fall, and so that will help as well. Yeah, so it’s not a full-time job. I do work on it a lot, every day at least.

There’s enough work to be done that it could fill a full-time job, but I won’t do that until it’s sustainable, which maybe someday it will be. That’d be kind of neat.

Mattias Geniar

I think one of the visible changes you did a couple of weeks ago was you, next to the download button of Caddy, I think you added a very big donation button just to say, this is free software. I’m making this for free. If you want help out, make a small donation without discussing any numbers.

How is the open source community? Are they supportive or are we all just free booters waiting to download the Caddy for free and never buy a beer for thanks?

Matt Holt

You know, it has been working out in the sense that like open sourcers, you know, if you’re an open source developer, you know how this works. And so I just, I don’t know, I just appeal to people’s goodwill. And again, like I’ve said before on this matter, it’s not a business plan, but it’s nice to be compensated a little bit for the time and the value it provides.

And it’s an option that’s available if you feel like that’s something that you want to do. But again, it’s not something I’m leaning on. I don’t have expectations.

But… it can be motivating. Like if I’ve been working on a really difficult bug or frustrating designed issue for a day or two, you know, and then I wake up to a notification that someone donated, it can be really motivating and just kind of be that kind of breath of fresh air. And I want to encourage people to do, to donate to any open source project, not just caddy, but any project that you benefit from that makes your life easier.

You know, give the developers a little bit of a push, a little bit of a high five, so to speak. And it’ll help motivate them, especially if they help you with like support issues or something.

Mattias Geniar

Exactly. I think if you mentioned it, it feels like a full-time job. I think for a lot of open source developers or developers that have projects to similar popularity as yours, it has to feel like it.

It’s not like you’re getting one request per week. I think your issue listing GitHub fills up daily. it’s either it’s a full-time or at least a daily job to keep that clean and to respond to questions if you do it weekly you probably can’t keep up and you wouldn’t have the the popularity the momentum that caddy has today i fully applaud the idea of supporting open source developers at the same time i think it’s a difficult one It puts the value that you’re getting out of it into the hands of whoever downloaded your software, who maybe can’t evaluate such software. I mean, if you were to go Microsoft style and package Caddy up in a big box, put it on a shelf in a store somewhere, how much would you charge?

It’s a very difficult question, I think. but open source is so underappreciated. I hope that one day you can make this your full-time job and be compensated fully for it. I truly hope that day ever comes.

At the same time, I’m not sure if the open source community is appreciative of such initiatives.

Matt Holt

You know, there are mixed opinions about it. I do believe that the open source community at large does appreciate the open source projects that they use. Even if they don’t make it known, they do appreciate it.

And I can tell, you know. But, yeah, it’s definitely a tricky issue because… I mean, if you run a project that is larger or is growing, if you want it to be a little more sustainable, the question is, well, do you go for a little bit of, I guess, compensation from a large portion of your user base, or do you go for a large compensation from just a few of your users?

Getting someone to donate $5 is very different than getting someone to donate $5,000. Oh, yeah. And so how you operate that and put that money to use is…

You know, those are different questions to answer for any open source project. I mean, ultimately I do it just because it’s fun and I love to see people using it and seeing their lives better. Well, I mean, their development, you know, their lives on the computer better because of it.

But yeah, those are all difficult questions.

Mattias Geniar

I think that what motivates most open source users is just to get a kick out of seeing how many people use it, how many problems you help solve. And at the same time, I can totally understand the little voice in your head saying, if I spend this much time developing a project, surely I should get something out of it. So yeah, I hope that, I think that, yeah, it’s just a difficult one.

Matt Holt

Yeah, because everyone’s situation is different. So between this December and this August, I’m right in the middle of it right now, but I’m between undergraduate and graduate school. So I haven’t opted to take on a full-time job right now just because it’s kind of an awkward timing in my life.

And that’s just my situation. So donations do help. um i have work of course to pay the bills but donations help and uh while while i will be a student i think that will be will be valuable and appreciated as well so but then when i’m done with school i can give back to the world and even i think on an even bigger scale or more i don’t know i can put that education to good use and give back more so it’s all just it all just depends indeed indeed

Mattias Geniar

Speaking of contributions, but perhaps in the other sense of the word, I’m sure you’ve received pull requests to caddy that may what add features or code that you perhaps didn’t want or that didn’t suit your style or your needs. How do you deal with that? How do you tell people that invested time and energy into contributing to your project that it’s just not good enough and you’re going to decline that pull request?

That seems like such a hard thing to do.

Matt Holt

Thank you.

Matt Holt

It is. When you know someone has put in effort, even if it’s a small change, they went ahead and took the initiative to make a change that can improve the project as far as their understanding goes. So we have an open discussion.

This is a skill that I’m still learning and will be trying to master my whole life. Because this is a people problem, right? Pull requests, they’re not about code as much as they are about people.

So we have an open discussion. I give Frank feedback, and I always try and give a thank you, of course. And most pull requests end up actually, at least with Caddy, they end up being accepted at some point or just deferred or closed temporarily, you know, until a certain phase is reached and then we can look at this again.

So some are just like a temporary closure and that kind of helps ease the blow a little bit. Others, I mean, I’ve had to close some that just aren’t in line with the philosophy of the project or would introduce a maintenance burden that I don’t want to personally assume and so we just talk about it and I just have to be open and frank and just very grateful but a lot of times pull requests can kind of be massaged into something that we can accept and so you just kind of have to work with it and be diplomatic and lay aside your pride and realize that the community they know their needs better than I know their needs

Mattias Geniar

But if they want to fill a very particular niche problem of caddy that you might not like to, like you mentioned, if they provide you code, their job is done, but you still have the maintenance burden. They may not be entirely up to it to keep maintaining that code for the years to come.

Matt Holt

So early on, after it was launched shortly, we started getting a lot of pull requests and feature requests for things that I didn’t believe lived in the web server at least, but it would be really useful to have. I could see that. So I redesigned a little bit of Caddy, and I’m actually still working on this right now, but you can extend Caddy and write add-ons. and so um so the first such add-on was a git add-on that that deploys your site with a git push really useful super awesome to have that built into your web server but at the same time not something that fits into caddy core you know that ships with everything with every to everyone so um our answer to meet people halfway, to still be able to accept their feature requests, the features they develop for Caddy, but not have to completely deny or refuse their pull requests.

And also, most importantly, to avoid sharding the project into all these different forks that have different features and have to be maintained. And then you get these old code bases that are out there running. To avoid all of that. you can submit an add-on and we can deploy it and make it available through Caddy’s download page.

Mattias Geniar

I think that’s, again, one of the best of both worlds scenarios. You still allow the contribution. At the same time, you avoid having to take responsibility for it.

That sounds a bit harsh, but it’s indeed a maintenance burden that you said. It’s not just a drive-by pull request that adds a feature that user X wants, but user Y will complain about. It separates it from the core of the project.

I think that’s a very good decision for long-term maintenance. yeah okay um what’s it what’s it been like been you’ve been running caddy for a year now two years how long has it been okay how is the community around it have you had any positive or negative experiences i think you’re interacting mostly with the go community i’m from a php background so go is still a um pretty obscure language to me i don’t know the community at heart

Matt Holt

You should get to know it. It actually came from the PHP community. And anyway, I found the Go community is great, too.

So the community has been fantastic. They’ve been growing, and I’ve learned so much from them. I’m learning more from this than anyone, I think.

It’s probably unfair. So it’s been a lot of fun to work on the project, to work with the people who are also interested in this and involved in this. It can be stressful and demanding at times.

I think part of that is just kind of a psychological illusion in myself. This isn’t a paying job. I don’t actually owe anyone anything, but I still want to deliver.

I still want to make great things out of this. So I just try to have fun. We just learn and we have a good time together.

People come and go, but we have some long timers who really are able to help a lot.

Mattias Geniar

I like the accidental word pun, we come and go. I think that’s indeed the case. It’s a community, it lives, it evolves.

I hope to see it still here for the years to come. So perhaps I should get involved in the Go community. But at the same time, PHP itself can already be a hassle, or perhaps not a hassle.

It can be intensive to keep up. Another community could just make it even more impossible to do.

Matt Holt

Well, here, maybe, actually, this reminds me, we can ease your transition, because I think just today, Laravel Valet, you used Laravel?

Mattias Geniar

I have used it in part, and I have been very curious about that mysterious tweet that you retweeted from Taylor, the creator of Laravel. So do tell.

Matt Holt

Yeah, so today he shipped Valet 1.1, I believe, which has Caddy built into it. So instead of using PHP’s built-in web server, it uses Caddy. And he says that the switch was to allow them to…

It does the same thing, but it will give them more capabilities in the future to… I don’t know. But Caddy is also a production-ready web server that…

So I imagine I actually don’t understand the choice fully. I agree and totally support it. I think this is a great use case.

Well, because Caddy is built, it is a great server for development as well. It’s kind of the point to make it easy. And so anyway, if you end up using Caddy to serve PHP and you want to improve it or whatever, feel free to dive in to go.

Mattias Geniar

well i think from um if we switch back to php and laravel i think taylor the creator of laravel i think he made a wise choice choosing caddy here because previously they shipped a vagrant machine with pre-installed software called homebrew which works fine it’s just vagrant up in the the project directory and it’ll get your virtualbox instance running and it will work but at the same time it’s quite the overhead it takes a bit of time for vagrant to boot it takes cpu and memory cycles for the linux machine to be emulated i think the idea of valet was to have a much simpler and faster way to develop and test applications on your mac or your windows or your linux I think Caddy fits in perfectly there for the simplest possible web server, yet at the same time giving enough flexibility to offer any kind of support that the developer might need. So I think it’s a very wise choice.

Matt Holt

Yeah, I do too. I was looking at it. Even Valet will even set up like a local host development domains like in your host file for you so that you can even load your development sites on a real domain and stuff.

So…

Mattias Geniar

it’s pretty cool and perhaps if this trend continues we’ll see we’ll start to see caddy files pop up in git repositories everywhere and soon we’ll just start to run our web servers like caddy instances sort of like simplified docker machines to get a configuration going Indeed. As a closing topic, you obviously develop Caddy. You spend a lot of your time in it.

What other open source tools do you think the listeners to this podcast should keep track of or know about or look into? What do you think is one of the coolest open source projects there is today?

Matt Holt

So I thought a lot about this because… Again, I don’t know the sysadmin community quite as well as I’d like. And so I can recommend a lot of the specific tools and projects I use.

For example, if you wanted to write Go using Sublime Text, I have a great list of open source projects that integrate really well there that I recommend. But to the wider community… You know, it’s kind of tricky.

Of course, this is a Go project, but there’s a project out there that I think almost any developer, any sysadmin could benefit from, and it’s called Goggs. And it’s the Go Git service. It’s written in Go, but it’s kind of like a GitHub replacement or a GitLab alternative. but it’s, again, it’s written in Go and very easy to set up and start running, like you might imagine.

It has a beautiful web interface. I think that’s a fantastic project. And Joe, who’s been, well, that’s his English name, he’s been working on that for at least a year and a half, two years now.

Mattias Geniar

and um it’s a fantastic project i saw that pop up i think last week or a couple of weeks ago um just by the screenshots of it um i can believe you when you say that he spent a year or a year and a half to uh working on it that’s probably not just the weekends because the uh the result that that delivered in that project is stunningly beautiful

Matt Holt

And, you know, he’s a student, or at least he was when I met him a couple years ago. So he’s brilliant. He’s busy.

He’s doing really good work.

Mattias Geniar

I get the feeling that most of the Go developers are students. They all deliver beautiful projects as far as I’m concerned. That’s great.

Okay, great. Unless you’d like to entice us with some more caddy goodness, I think we can wrap up here.

Matt Holt

I have nothing to add at this point.

Mattias Geniar

Matt, if listeners wanted to catch up with you, how could they find you online?

Matt Holt

You can follow me on Twitter. I’m at mholt6. And that would probably be the best way.

Mattias Geniar

And the Caddy web server lives at which address?

Matt Holt

Oh, at caddyserver.com. And that’s, of course, with an HTTPS. Obviously.

Mattias Geniar

I’ll add all those links in the show notes. So if people want to click through, they can. Okay, Matt, thanks a lot for your time and the hectic schedule to get this going.

No problem. This was the first recording for me. I’m very happy that you were the very first guest on this podcast because it’s been really interesting for me.

And I hope I can keep this trend going because if all of my guests are like you, this shouldn’t be a problem.

Matt Holt

Yeah, well, I hope it’ll be. This is a great podcast. I hope it’ll be a success.

Mattias Geniar

Thank you very much. Okay, Matt, take care.

Matt Holt

Yeah, good talking to you.

Mattias Geniar

Bye.