A couple of days ago I was debugging a particularly interesting problem with PHP’s sessions being stored in a Memcached instance. More specifically, this system was running the Magento eCommerce software.
The problem manifested itself with working PHP sessions for the “frontend”, the normal users that log in and do some shopping. However, for all backend admins, when they wanted to log in they got a PHP error message:
Warning: Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (127.0.0.1:11211) in Unknown on line 0
Since sessions in Memcached worked for normal users, I thought I could rule out the general PHP config that stores things in Memcached. Just for completeness sake, here it is:
memcache.allow_failover = 1 memcache.chunk_size = 32768 memcache.default_port = 11211 memcache.hash_function = crc32 session.save_handler = memcached session.save_path = 127.0.0.1:11211
Pretty standard.
What was happening to those Magento admin sessions causing the fatal PHP error then?
Turns out, the default Memcached instance you set up (yum install memcached) has a default maximum key size of 1MB. This key/value size is refered to as a “slabsize” in Memcached.
On a CentOS machine, the default Memcached config looks like this.
$ cat /etc/sysconfig/memcached PORT="11211" USER="memcached" MAXCONN="1024" CACHESIZE="64" OPTIONS=""
A safe addition to this default config would be to let Memcached only listen to localhost, instead of all ports (which it does by default). We’ll fix that later on, too.
The default options set a cache size of 64MB but leave the maximum key size of 1MB intact. An admin session in Magento can, apparently, exceed that 1MB size.
If a PHP session exceeds the maximum Memcached key/slab size, it will throw a fatal PHP error that it failed to write the session data.
That explains the following error:
Warning: Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (127.0.0.1:11211) in Unknown on line 0
Now to fix it, you can modify your Memcached configuration and add an option to increase the maximum key/slab size in Memcached.
A typical Memcached instance runs like this:
$ memcached -d -p 11211 -u memcached -m 64 -c 1024 -P /var/run/memcached/memcached.pid
We’ll modify the config to support only listening on localhost and with a maximum key/slab size of 10MB, instead of the default 1MB.
$ cat /etc/sysconfig/memcached PORT="11211" USER="memcached" MAXCONN="1024" CACHESIZE="64" OPTIONS="-l 127.0.0.1 -I 10m"
In particular, we’ve modified the OPTIONS= block and added the -l (lower case L) and -I (upper case i) to that list. The -I increases the maximum key/slab size in Memcached, as explained by the manual.
-I <size> Override the default size of each slab page. Default is 1mb. Default is 1m, minimum is 1k, max is 128m. Adjusting this value changes the item size limit. Beware that this also increases the number of slabs (use -v to view), and the overal memory usage of memcached.
If we restart the memcached instance to activate our configs and look at the running instance again, we’ll see the difference.
$ /etc/init.d/memcached restart $ memcached -d -p 11211 -u memcached -m 64 -c 1024 -P /var/run/memcached/memcached.pid -l 127.0.0.1 -I 10
If you restart Memcached, you’ll get a friendly heads-up that increasing the key/slab size also increases the minimum memory requirements:
$/etc/init.d/memcached restart
Stopping memcached: [ OK ]
Starting memcached: WARNING: Setting item max size above 1MB is not recommended!
Raising this limit increases the minimum memory requirements
and will decrease your memory efficiency.
[ OK ]
But, if your CMS needs bigger session data and you have no way to change it, you have no choice.
If you want to have a look at what data is currently in Memcached, read this blogpost: Read all keys & values from a Memcached instance.