Drupal’s itok parameters explained and a Varnish workaround for different backends

Ever since the Drupal 7.20 update, a new system was introduced to allow the generation of thumbnails. Instead of linking directly to an image, the URLs are extended with a ?itok=$random_string parameter. So, how does this work?

This may very well be outdated once Drupal 8 comes out, but I couldn’t figure it out straight away on Drupal 7.x, so here goes.

The feature got introduced in the 7.20 release, a security release from 2013. The issue that got resolved was a denial of service attack in Drupal’s core, related to how derivative images (like thumbnails) got generated. The fix was adding the ?itok parameter and all logic involved with processing it.

The calculation of the itok parameter goes like this.

define('IMAGE_DERIVATIVE_TOKEN', 'itok');

function image_style_path_token($style_name, $uri) {
  // Return the first eight characters.
  return substr ( drupal_hmac_base64 ( $style_name . ':' . $uri, drupal_get_private_key() . drupal_get_hash_salt() ), 0, 8);
}

function function image_style_url($style_name, $path) {
   $token_query = array(IMAGE_DERIVATIVE_TOKEN => image_style_path_token($style_name, $path));

   return url($directory_path . '/' . file_uri_target($uri), array('absolute' => TRUE, 'query' => $token_query));
}

I’ve changed the ordering of the PHP logic to be more readable, but it pretty much boils down to the diff in commit 750aabb.

The function gets called with a $style_name parameter, which in most cases has the value ‘thumbnail’ and an $image_uri parameter, which is the full URL to the image to be displayed.

$image_info['path'] = image_style_url('thumbnail', $image_uri);

So every request to a page made in Drupal can trigger that piece of logic, which will:

1. Generate a unique token, based on: Drupal’s private key and the URL to the file.
2. Append that to the image URL using the ?itok=$token parameter.
3. The server receives that URL, knows it’s validated (because Drupal’s unique private key was used) and is allowed to generate the derivative image.
4. If the derivative image already exists, it’s just served as-is and the ?itok parameter gets ignored.

Drupal’s Private Key gets called with the drupal_get_private_key() function and that value comes from the database;

MySQL > select * from variable where name='drupal_private_key';
+--------------------+-----------------------------------------------------+
| name               | value                                               |
+--------------------+-----------------------------------------------------+
| drupal_private_key | s:43:"MelQl73fQ1rQfNLAondL-tLE4J1q5AlhLidA1glTjFd"; |
+--------------------+-----------------------------------------------------+

So the end result is a unique token file per full URL.

This technique can cause a bit of a headache when used with Varnish caches. If you have logic in your Varnish code that sets a different backend based on file extension (ie: to let static content be served by a different webserver) the thumbnails aren’t automatically generated.

vcl_recv {
  # Always let the static backend process the files
  if (req.url ~ "(?i)\.(gif|jpeg|jpg|png)(\?[a-z0-9=\.]+)?$") {
     # This could be a backend like Nginx, Lighttpd, ... optimised for static content
     # Beware; this regex only catches a couple of image extensions, add more if needed!
     set req.backend = static_servers;
  }
}

But if that triggers a 404, you’ll want the request to be sent to your PHP servers so they can generate a proper thumbnail. After all, the “static” backend probably can’t generate PHP code and allow the thumbnail to be created.

Here’s a workaround you can use for this particular use case, by using the restart function in Varnish 3. If you’re using Varnish 4.x, this won’t work and you’ll have to research a bit more – sorry about that.

vcl_recv {
   # Always let the static backend process the files
   if (req.url ~ "(?i)\.(gif|jpeg|jpg|png)(\?[a-z0-9=\.]+)?$") {
     # Beware; this regex only catches a couple of image extensions, add more if needed!

     if (req.restarts == 1) {
        # A restart was triggered from vcl_fetch: force a PHP request to generate the thumbnail
        # We only set the backend to the PHP servers on the first restart
        set req.backend = php_servers;
      } else {
        # This could be a backend like Nginx, Lighttpd, ... optimised for static content
        set req.backend = static_servers;
      }
   }
}

vcl_fetch {
   if (req.url ~ "\?itok=" && beresp.status == 404) {
      return (restart);
   }
}

That should solve the issue of sending requests to a particular backend, but keeping the ability to redirect to a “PHP backend” whenever a thumbnail needs to be generated.