Varnish running in foreground but fails to run as service/daemon

Mattias Geniar, Monday, November 28, 2011

I had an annoying problem last week that turned out to be my own silly mistake. But since I couldn't quite find a lot of information on it, I'll sum it up here so I'll never forget it again.

If I started a Varnish service in foreground mode, it worked like a charm -- if I tried it via the "service varnish start" or the init.d script, it kept failing. Stracing couldn't quite tell me what I was missing at first.

# varnishd -F -f /usr/local/etc/varnish/default.vcl
socket(): Address family not supported by protocol child (5214)
Started Child (5214) said Child starts Child (5214) said SMF.s0
mmap'ed 104857600 bytes of 104857600

^CManager got SIGINT => CTRL+C

killed Stopping Child

So foreground runs like a charm.

Yet the service kept failing on me.

# service varnish start
Starting Varnish Cache:                                    [FAILED]

And when stracing, there's no real clue as to why it failed.

# strace -e trace=file service varnish star
execve("/sbin/service",
["service", "varnish", "start"], [/* 18 vars */]) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib64/libtinfo.so.5", O_RDONLY) = 3
open("/lib64/libdl.so.2", O_RDONLY) = 3
open("/lib64/libc.so.6", O_RDONLY) = 3
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
open("/proc/meminfo", O_RDONLY) = 3
stat("/root", {st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0 stat(".",
{st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
open("/sbin/service", O_RDONLY) = 3
open("/etc/init.d/functions", O_RDONLY) = 3
--- SIGCHLD (Child exited) @ 0 (0) --- stat("/etc/sysconfig/i18n",
{st_mode=S_IFREG|0644, st_size=47, ...}) = 0 open("/dev/null",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 open("/etc/profile.d/lang.sh",
O_RDONLY) = 3
stat("/root/.i18n", 0x7fffa8819680) = -1 ENOENT (No such file or directory)
stat("/etc/sysconfig/init", {st_mode=S_IFREG|0644, st_size=1153, ...}) = 0
open("/etc/sysconfig/init", O_RDONLY) = 3
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/initscripts.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/initscripts.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/initscripts.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
--- SIGCHLD (Child exited) @ 0 (0) ---
--- SIGCHLD (Child exited) @ 0 (0) ---
chdir("/") = 0
stat("/etc/init.d/varnish", {st_mode=S_IFREG|0755, st_size=2851, ...})
= 0 stat(".", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
stat("/sbin/env", 0x7fffa881a600) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/env", 0x7fffa881a600) = -1 ENOENT (No such file or directory)
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", X_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", R_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
access("/bin/env", X_OK) = 0
stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
vaccess("/bin/env", R_OK) = 0
Starting Varnish Cache: [FAILED]
--- SIGCHLD (Child exited) @ 0 (0) ---

What the strace doesn't show, is that when you start Varnish as a service, it reads an additional config file: /etc/sysconfig/varnish on CentOS/RHEL or /etc/default/varnish on Ubuntu/Debian systems. Chances are, if you can run Varnish in the foreground but not as a service, and you're referring to the exact same VCLs, your problem is somewhere in the default configuration files that determine your service.

My problem turned out to be a missing directive in /etc/sysconfig/varnish that I overlooked.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. Currently working on DNS Spy. Follow me on Twitter as @mattiasgeniar.

I respect your privacy and you won't get spam. Ever.
Just a weekly newsletter about Linux and open source.

SysCast podcast

In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!

cron.weekly newsletter

A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Safwan Friday, April 5, 2013 at 02:54 (permalink)

Hi,

I’m having the same problem. Could you tell me what ‘directive’ is missing from /etc/sysconfig/varnish… I tried everything and you seem to be the only person with the same problem.

Reply


    Mattias Geniar Friday, April 12, 2013 at 17:55 (permalink)

    Hi Safwan,

    Apologies for the late reply, you may have already fixed it. Here’s an example config that works:

    $ cat /etc/sysconfig/varnish | grep -v '^#'
    NFILES=131072
    MEMLOCK=82000
    RELOAD_VCL=1
    VARNISH_VCL_CONF=/etc/varnish/default.vcl
    VARNISH_LISTEN_PORT=80
    VARNISH_ADMIN_LISTEN_ADDRESS=0.0.0.0
    VARNISH_ADMIN_LISTEN_PORT=6082
    VARNISH_MIN_THREADS=1
    VARNISH_MAX_THREADS=5000
    VARNISH_THREAD_TIMEOUT=120
    VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
    VARNISH_STORAGE_SIZE=3G
    KINEPOLIS_VARNISH_STORAGE="file,/var/lib/varnish/varnish_storage.bin,3G"
    KINEPOLIS_CONNECT_TIMEOUT="connect_timeout=600"
    VARNISH_TTL=900
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
                 -f ${VARNISH_VCL_CONF} \
                 -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
                 -t ${VARNISH_TTL} \
                 -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
                 -u varnish -g varnish \
                 -s ${KINEPOLIS_VARNISH_STORAGE} \
                 -p ${KINEPOLIS_CONNECT_TIMEOUT} \
                    -p sess_workspace=262144 \
                    -p thread_pools=2 \
                    -p thread_pool_min=200 \
                    -p thread_pool_max=5000 \
                    -p thread_pool_add_delay=2 \
                    -p http_req_size=131072 \
                    -p http_req_hdr_len=49152 "
    

    Reply


    Martin Wednesday, February 24, 2016 at 18:02 (permalink)

    You should not have duplicate entries for localhost in /etc/hosts.

    Try

    getent hosts localhost

    and you will see that only the last entry is used. Remove localhost host from ::1.

    Reply


Leave a Reply

Your email address will not be published. Required fields are marked *