Varnish: same hash, different results? Check the Vary header!

Mattias Geniar, Tuesday, April 3, 2018

I'll admit I get bitten by the Vary header once every few months. It's something a lot of CMS's randomly add, and it has a serious impact on how Varnish handles and treats requests.

For instance, here's a request I was troubleshooting that had these varnishlog hash() data:

-   VCL_call       HASH
-   Hash           "/images/path/to/file.jpg%00"
-   Hash           "http%00"
-   Hash           "www.yoursite.tld%00"
-   Hash           "/images/path/to/file.jpg.jpg%00"
-   Hash           "www.yoursite.tld%00"
-   VCL_return     lookup
-   VCL_call       MISS

A new request, giving the exact same hashing data, would return a different page from the cache/backend. So why does a request with the same hash return different data?

Let me introduce the Vary header.

In this case, the page I was requesting added the following header:

Vary: Accept-Encoding,User-Agent

This instructs Varnish to keep a separate version each page for every value of Accept-Encoding and User-Agent it finds.

The Accept-Encoding would make sense, but Varnish already handles that internally. A gziped/plain version will return different data, that makes sense. There's no real point in adding that header for Varnish, but other proxies in between might still benefit from it.

The User-Agent is plain nonsense, why would you serve a different version of a page per browser? If you consider a typical User-Agent string to contain text like Mozilla/5.0 (Macintosh; Intel Mac OS X...) AppleWebKit/537.xx (KHTML, like Gecko) Chrome/65.x.y.z Safari/xxx, that's practically unique per visitor you have.

So, quick hack in this case, I remove the Vary header altogether.

sub vcl_backend_response {
  unset beresp.http.Vary;
  ...
}

No more variations of the cache based on what a random CMS does or says.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Robert Sander Wednesday, April 4, 2018 at 07:42 - Reply

You can use the User-Agent header to detect mobile users and serve your mobile content for them. In this case it is important for caches to store different results based on that header.


    Mattias Geniar Wednesday, April 4, 2018 at 10:44 - Reply

    You can use the User-Agent header to detect mobile users and serve your mobile content for them. In this case it is important for caches to store different results based on that header.

    In that case you would still normalize the header, having each version of Chrome (mobile or desktop) be served a different page in the cache is overkill.

    Dummy code like this would give you much better caching hits and only keep 2 versions of the page. You can’t do this in the Vary header though, as it requires normalization of the User-Agent.

    sub vcl_recv {
      if (req.http.User-Agent ~ "(?i)ip(hone|od)") {
        set req.http.X-Device = "smartphone";
      } else {
        set req.http.X-Device = "desktop";
      }
      ...
    }
    
    sub vcl_hash {
      if (req.http.X-Device) {
        hash_data(req.http.X-Device);
      }
    }
    

    There’s a pretty good varnish-devicedetect VCL repo that contains a lot of that mobile detection logic.


Leave a Reply

Your email address will not be published. Required fields are marked *