Varnish running in foreground but fails to run as service/daemon

Mattias Geniar, Monday, November 28, 2011

I had an annoying problem last week that turned out to be my own silly mistake. But since I couldn't quite find a lot of information on it, I'll sum it up here so I'll never forget it again.

If I started a Varnish service in foreground mode, it worked like a charm -- if I tried it via the "service varnish start" or the init.d script, it kept failing. Stracing couldn't quite tell me what I was missing at first.

# varnishd -F -f /usr/local/etc/varnish/default.vcl
socket(): Address family not supported by protocol child (5214)
Started Child (5214) said Child starts Child (5214) said SMF.s0
mmap'ed 104857600 bytes of 104857600

^CManager got SIGINT => CTRL+C

killed Stopping Child

So foreground runs like a charm.

Yet the service kept failing on me.

# service varnish start
Starting Varnish Cache:                                    [FAILED]

And when stracing, there's no real clue as to why it failed.

# strace -e trace=file service varnish star
execve("/sbin/service",
["service", "varnish", "start"], [/* 18 vars */]) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib64/libtinfo.so.5", O_RDONLY) = 3
open("/lib64/libdl.so.2", O_RDONLY) = 3
open("/lib64/libc.so.6", O_RDONLY) = 3
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
open("/proc/meminfo", O_RDONLY) = 3
stat("/root", {st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0 stat(".",
{st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
open("/sbin/service", O_RDONLY) = 3
open("/etc/init.d/functions", O_RDONLY) = 3
--- SIGCHLD (Child exited) @ 0 (0) --- stat("/etc/sysconfig/i18n",
{st_mode=S_IFREG|0644, st_size=47, ...}) = 0 open("/dev/null",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 open("/etc/profile.d/lang.sh",
O_RDONLY) = 3
stat("/root/.i18n", 0x7fffa8819680) = -1 ENOENT (No such file or directory)
stat("/etc/sysconfig/init", {st_mode=S_IFREG|0644, st_size=1153, ...}) = 0
open("/etc/sysconfig/init", O_RDONLY) = 3
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/initscripts.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/initscripts.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/initscripts.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
--- SIGCHLD (Child exited) @ 0 (0) ---
--- SIGCHLD (Child exited) @ 0 (0) ---
chdir("/") = 0
stat("/etc/init.d/varnish", {st_mode=S_IFREG|0755, st_size=2851, ...})
= 0 stat(".", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
stat("/sbin/env", 0x7fffa881a600) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/env", 0x7fffa881a600) = -1 ENOENT (No such file or directory)
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", X_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", R_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", X_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
vaccess("/bin/env", R_OK) = 0
Starting Varnish Cache: [FAILED]
--- SIGCHLD (Child exited) @ 0 (0) ---

What the strace doesn't show, is that when you start Varnish as a service, it reads an additional config file: /etc/sysconfig/varnish on CentOS/RHEL or /etc/default/varnish on Ubuntu/Debian systems. Chances are, if you can run Varnish in the foreground but not as a service, and you're referring to the exact same VCLs, your problem is somewhere in the default configuration files that determine your service.

My problem turned out to be a missing directive in /etc/sysconfig/varnish that I overlooked.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.


Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Safwan Friday, April 5, 2013 at 02:54 - Reply

Hi,

I’m having the same problem. Could you tell me what ‘directive’ is missing from /etc/sysconfig/varnish… I tried everything and you seem to be the only person with the same problem.


    Mattias Geniar Friday, April 12, 2013 at 17:55 - Reply

    Hi Safwan,

    Apologies for the late reply, you may have already fixed it. Here’s an example config that works:

    $ cat /etc/sysconfig/varnish | grep -v '^#'
    NFILES=131072
    MEMLOCK=82000
    RELOAD_VCL=1
    VARNISH_VCL_CONF=/etc/varnish/default.vcl
    VARNISH_LISTEN_PORT=80
    VARNISH_ADMIN_LISTEN_ADDRESS=0.0.0.0
    VARNISH_ADMIN_LISTEN_PORT=6082
    VARNISH_MIN_THREADS=1
    VARNISH_MAX_THREADS=5000
    VARNISH_THREAD_TIMEOUT=120
    VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
    VARNISH_STORAGE_SIZE=3G
    KINEPOLIS_VARNISH_STORAGE="file,/var/lib/varnish/varnish_storage.bin,3G"
    KINEPOLIS_CONNECT_TIMEOUT="connect_timeout=600"
    VARNISH_TTL=900
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
                 -f ${VARNISH_VCL_CONF} \
                 -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
                 -t ${VARNISH_TTL} \
                 -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
                 -u varnish -g varnish \
                 -s ${KINEPOLIS_VARNISH_STORAGE} \
                 -p ${KINEPOLIS_CONNECT_TIMEOUT} \
                    -p sess_workspace=262144 \
                    -p thread_pools=2 \
                    -p thread_pool_min=200 \
                    -p thread_pool_max=5000 \
                    -p thread_pool_add_delay=2 \
                    -p http_req_size=131072 \
                    -p http_req_hdr_len=49152 "
    

    Martin Wednesday, February 24, 2016 at 18:02 - Reply

    You should not have duplicate entries for localhost in /etc/hosts.

    Try

    getent hosts localhost

    and you will see that only the last entry is used. Remove localhost host from ::1.


Leave a Reply

Your email address will not be published. Required fields are marked *