Using Oh Dear to keep your Varnish cache warm

This post was originally published on the Oh Dear blog on 8 April 2019. I moved it to my own blog in October 2026.

Slow websites are annoying, right? We sure think so.

One common solution is to introduce a caching proxy like Varnish to help cache pages and reduce your server load.

The good news is, if you have Oh Dear, you can let those 2 work together.

Varnish + the Oh Dear crawler#

The idea is as follows: if you’ve enabled our broken links or mixed content checks for any of your sites, we’ll crawl your sites to find any broken pages.

On top of that, you can set custom HTTP headers per monitor that get added to our requests. The uptime check has its own set of headers, and the crawler (broken links, mixed content and sitemap checks) has another. So you can give each one a different secret.

Combining our crawler and the custom HTTP headers allows you to authorize our crawler in your Varnish configs to let it update the cache.

Configuring Varnish to refresh your cache when Oh Dear crawls your site#

There are several default Varnish configs available you can use as the boilerplate for your Varnish service. One of them is maintained by one of the co-founders of Oh Dear (hi!) and can be found here: Varnish 6 configuration template . For a VCL that compiles on today’s 9.x releases, start from the one in my 2026 Varnish post .

Either way, it’s relatively easy to allow Oh Dear to update your cached pages.

First, add this to your vcl_recv routine.

sub vcl_recv {
    if (req.http.OhDear-Authorized-Request == "ASecretStringOrPassphrase") {
        set req.hash_always_miss = true;
    }

    # ... the rest of your vcl_recv
}

Now, every time Varnish receives a request with the header OhDear-Authorized-Request that has the value ASecretStringOrPassphrase, it will refresh the current page from its backend (aka: that request won’t be served from the cache, it goes to your backend server). The fresh response replaces the cached object, so the next visitor gets the new version.

In Oh Dear, you add this header in the Request headers section of your broken links (or mixed content) check settings. From then on, that extra header will be present every time we crawl your pages.

Our crawler also identifies itself with this user agent:

OhDear.app (+https://ohdear.app/docs/features/broken-links-detection)

That’s handy to spot us in your access logs, but don’t use it to grant anything. Anyone can send that user agent. The secret header is what proves the request came from your Oh Dear account.

This also allows extra possibilities in your Varnish code, like authorizing our crawler to access otherwise limited pages.

Checking the uptime behind your cache#

Using similar logic, we can also allow the uptime checks to bypass the cache altogether to check the availability of your site.

It wouldn’t be the first time a site appears online, because the monitoring probes were answered by a cached response. Your users that bypass the cache (they might be logged in) are still seeing your error pages.

To let Oh Dear bypass the cache every time without touching the TTL (Time-To-Live) or currently cached pages, you can use logic similar to this.

sub vcl_recv {
    if (req.http.OhDear-Authorized-Request == "ASecretStringOrPassphrase") {
        return (pass);
    }
}

The uptime check takes its headers from the Headers tab in its own request settings, separate from the crawler’s. Use a different value there than for the crawler and you can tell the two apart in your VCL: one refreshes the cache, the other skips it.

I tested both snippets on Varnish Cache 9.1.0 (the varnish:latest Docker image) with varnishd -C and a backend that stamps every response with its own timestamp. With hash_always_miss, the request carrying the header got a fresh backend response and the next normal request was served that fresh copy from the cache. With return (pass), the request carrying the header went to the backend and the cached copy stayed untouched for everyone else.

If you’re running Varnish on systemd, my notes on drop-in overrides instead of varnish.params cover the service side, and debugging cookie stripping helps when a page stubbornly refuses to cache.

We’re certain there are other use cases out there that we haven’t even thought of. If you’re doing something similar, I’d love to hear about it!