Increase the number of open files for jobs managed by supervisord

Mattias Geniar, Thursday, November 29, 2018

In Linux, a non-privileged user by default can only open 1024 files on a machine. This includes handles to log files, but also local sockets, TCP ports, ... everything's a file and the usage is limited, as a system protection.

Normally, we can increase the amount of processes a particular user can open by increasing the system limits. This is configured in /etc/security/limits.d/.

For instance, this allows the user john to open up to 10.000 files.

$ cat /etc/security/limits.d/john.conf
john		soft		nofile		10000

You would assume that once configured, this would apply to all the commands that run as the user john. Alas, that's not the case if you use supervisord to run a process.

Take the following supervisor job for instance:

$ cat /etc/supervisord.d/john.ini
[john]
command=/usr/bin/php /path/to/script.php
numprocs=1
autostart=true
autorestart=true
user=john

This would add a job to supervisor to always keep the task /usr/bin/php /path/to/script.php running as the user john, and if it were to crash or stop, it would automatically restart it.

However, if we were to inspect the actual limits being enforced on that process, we'd find the following.

$ cat /proc/19153/limits
Limit                     Soft Limit           Hard Limit           Units
...
Max open files            1024                 4096                 files

The process has a soft-limit of 1024 files and a hard limit of 4096, despite an increase in the amount of files it can open in our limits.d directory.

The reason is that supervisord has a setting of its own, minfds, that it uses to set the amount of files it can open. And that setting gets inherited by all the children that supervisord spawns, so it overrides any setting you may set in limits.d.

Its default value is set to 1024 and can be increased so anything you'd like (or need).

$ cat /etc/supervisord.conf
[supervisord]
...
minfds=1024;

You'll find this file on /etc/supervisor/supervisord.conf on Debian or Ubuntu systems. Either add or modify the minfds parameter, restart supervisord (which will restart all your spawned jobs, too) and you'll notice the limits have actually been increased.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Inbound links