In Linux, a non-privileged user by default can only open 1024 files on a machine. This includes handles to log files, but also local sockets, TCP ports, … everything’s a file and the usage is limited, as a system protection.
Normally, we can increase the amount of processes a particular user can open by increasing the system limits. This is configured in /etc/security/limits.d/.
For instance, this allows the user john to open up to 10.000 files.
$ cat /etc/security/limits.d/john.conf john soft nofile 10000
You would assume that once configured, this would apply to all the commands that run as the user john. Alas, that’s not the case if you use supervisord to run a process.
Take the following supervisor job for instance:
$ cat /etc/supervisord.d/john.ini [john] command=/usr/bin/php /path/to/script.php numprocs=1 autostart=true autorestart=true user=john
This would add a job to supervisor to always keep the task /usr/bin/php /path/to/script.php running as the user john, and if it were to crash or stop, it would automatically restart it.
However, if we were to inspect the actual limits being enforced on that process, we’d find the following.
$ cat /proc/19153/limits Limit Soft Limit Hard Limit Units ... Max open files 1024 4096 files
The process has a soft-limit of 1024 files and a hard limit of 4096, despite an increase in the amount of files it can open in our limits.d directory.
The reason is that supervisord has a setting of its own, minfds, that it uses to set the amount of files it can open. And that setting gets inherited by all the children that supervisord spawns, so it overrides any setting you may set in limits.d.
Its default value is set to 1024 and can be increased so anything you’d like (or need).
$ cat /etc/supervisord.conf [supervisord] ... minfds=1024;
You’ll find this file on /etc/supervisor/supervisord.conf on Debian or Ubuntu systems. Either add or modify the minfds parameter, restart supervisord (which will restart all your spawned jobs, too) and you’ll notice the limits have actually been increased.