The “.well-known” directory on webservers (aka: RFC 5785)

I first came across the concept of the directory named '.well-known' when automating Let's Encrypt, the free SSL certificate authority. It didn't strike me as abnormal to have a validation happen via an HTTP or HTTPS GET request. Those Let's Encrypt validation URLs usually point to site.tld/.well-known/acme-challenge/random-key.txt.

At first I thought this was just the random URL used by Let's Encrypt, but today I learned there's more to it. Allow me to introduce RFC-5785.

It is increasingly common for Web-based protocols to require the
discovery of policy or other information about a host ("site-wide
metadata") before making a request.

[...]

When this happens, it is common to designate a "well-known location"
for such data, so that it can be easily located.

[...]

To address this, this memo defines a path prefix in HTTP(S) URIs for
these "well-known locations", "/.well-known/".

RFC-5785

The directory location /.well-known isn't a coincidence, it's the result of a carefully considered RFC. This directory can be used for all kinds of information discovery.

For instance, Let's Encrypt uses a subdirectory called /.well-known/acme-challenge/, where 'ACME' stands for Automated Certificate Management Environment.

There's a Do Not Track policy (DNT) that uses a similar file in that directory for validation: /.well-known/dnt-policy.txt.

There are a couple of webservers configurations that prevent opening directories that start with dot ".". The reasoning behind it is that it might give away sensitive information, like a .git or .svn directory (which probably shouldn't even be on your webserver in the first place).

However, this RFC for the .well-known directory make it clear that at least this particular directory should be allowed to be accessed (not in a directory-listing sense, but for direct file access).