Advanced Monitoring Of HAProxy With Zabbix Agent

Tired of the privacy invasion of browsers? Worried about the risk of seeing ads everywhere? Give the Brave Browser a try. All Chrome extensions are compatible. None of the telemetry. Auto-blocks ads and still support content creators like me.

Download the Brave browser now »

Profile image of Mattias Geniar

Mattias Geniar, October 22, 2010

Follow me on Twitter as @mattiasgeniar

If you use HAProxy as a load balancing tool, and Zabbix as your monitoring tool of choice (which you should use both, really – I love ‘m), you can use the following template/guide to do some advanced logging of HAProxy’s usage.

It’s based on the same techniques used by hatop to use a “top”-like interface to check your HAProxy setup.

Enable statistics in HAProxy

For starters, we’ll need to modify HAProxy ‘s config, to include the “stats socket” parameter. Add this to the “global” section of the HAProxy config file.

global
# Create a socket that allows us to query it directly
stats socket    /tmp/haproxy.sock

Install ‘socat’ for querying the socket

As discussed in the HAProxy documentation (see section “9.2. Unix Socket commands"), we can query this socket directly and ask it for usage statistics. To do so, use a tool like socat.

$ cd /usr/local/src
$ wget http://www.dest-unreach.org/socat/download/socat-1.7.1.3.tar.gz
$ tar xzf socat-1.7.1.3.tar.gz
$ cd socat-1.7.1.3/
$ ./configure
$ make
$ make install

Here’s how you can use it.

$ echo "help" | socat /tmp/haproxy.sock stdio 
clear counters : clear max statistics counters (add 'all' for all counters)
help           : this message
prompt         : toggle interactive mode with prompt
quit           : disconnect
show info      : report information about the running process
show stat      : report counters for each proxy and server
show errors    : report last request and response errors for each proxy
show sess [id] : report the list of current sessions or dump this session
get weight     : report a server's current weight
set weight     : change a server's weight
set timeout    : change a timeout setting
disable server : set a server in maintenance mode
enable server  : re-enable a server that was previously in maintenance mode

Querying useful statistics from HAProxy

And some example output of various (interesting) commands: “show info

$ echo "show info" | socat /tmp/haproxy.sock stdio
Name: HAProxy
Version: 1.4.8
Release_date: 2010/06/16
Nbproc: 1
Process_num: 1
Pid: 3940
Uptime: 0d 0h14m11s
Uptime_sec: 851
Memmax_MB: 0
Ulimit-n: 4112
Maxsock: 4112
Maxconn: 2048
Maxpipes: 0
CurrConns: 3
PipesUsed: 0
PipesFree: 0
Tasks: 6
Run_queue: 1
node: lb02.lab.mojah.be
description:

“show sess”: this will dump all known sessions. The following example only has 2 active sessions. Using this on high-traffic systems will be terribly slow, as it’ll show _all_ active sessions. I don’t recommend using it, unless absolutely necessary.

$ echo "show sess" | socat /tmp/haproxy.sock stdio
0x811c820: proto=unix_stream ts=09 age=0s calls=2 rq[f=d0d220h,l=0,an=00h,rx=,wx=,ax=] rp[f=009000h,l=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[7,0h,fd=-1,ex=] exp=10s

“show stat”: will show all statistics in CSV format.

$ echo "show stat" | socat /tmp/haproxy.sock stdio 
# pxname, svname, qcur, qmax, scur, smax, slim, stot, bin, bout, dreq, dresp, ereq, econ, eresp, wretr, wredis, status, weight, act, bck, chkfail, chkdown, lastchg, downtime,qlimit, pid, iid, sid, throttle, lbtot, tracked, type, rate,rate_lim, rate_max, check_status, check_code, check_duration, hrsp_1xx, hrsp_2xx, hrsp_3xx, hrsp_4xx, hrsp_5xx, hrsp_other, hanafail, req_rate, req_rate_max, req_tot, cli_abrt, srv_abrt,
stats,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,
stats,BACKEND,0,0,0,0,2000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,1023,0,,1,1,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,
smtp,FRONTEND,,,0,2,2000,3,0,82,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,
smtp,smtp1,0,0,0,1,,1,0,44,,0,,0,0,0,0,UP,1,1,0,0,1,228,793,,1,2,1,,1,,2,0,,1,L4OK,,0,,,,,,,0,,,,1,0,
smtp,smtp2,0,0,0,1,,1,0,38,,0,,0,0,0,0,UP,1,1,0,0,1,230,791,,1,2,2,,1,,2,0,,1,L4OK,,0,,,,,,,0,,,,1,0,
smtp,smtp3,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1020,1020,,1,2,3,,0,,2,0,,0,L4TOUT,,2000,,,,,,,0,,,,0,0,
smtp,BACKEND,0,0,0,2,2000,3,0,82,0,0,,1,0,0,0,UP,2,2,0,,1,230,790,,1,2,0,,2,,1,0,,1,,,,,,,,,,,,,,2,0,

For instance, in my setup I have 3 listening SMTP servers (smtp1, smtp2 and smpt3 above) in the frontend. I can query the socket, to tell me how many current sessions I have. I can filter for totals “smtp,FRONTEND” or per server “smtp,smtp1”.

$ echo "show stat" | socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $5}'
1

I’m piping the output of my socket-query to grep, to only match the server I want, and I convert the CSV output to a space-seperated output via sed, so I can more easily extract the column I want via awk. If you change the “print $5” line, you can choose to print other columns as well. Here are the more interesting ones.

print $3: current queued requests (qcur)
print $4: max queued requests (qmax)
print $26: queue limited (qlimit)
print $5: current sessions (scur)
print $6: max sessions (smax)
print $14: connection errors (econ)
print $37: status of last health check (should be L4OK, L6OK, L6OK (check_status)

Do note that if you’re grepping for the FRONTEND or BACKEND groups, there will be no output for qcur and qmax, which means all numbers will shift 2 places to left. Meaning scur is not $5, but $3 – as awk won’t count that as an occurence.

Custom UserParameters in Zabbix for HAProxy stats

You can use the following “template” on the HAProxy server to use Zabbix Agent checks and retrieve this data. Be sure to modify the file, so it greps for your correct servers!

Place this file in /etc/zabbix/zabbix_agent.d/haproxy_totals.conf (and make sure the ‘Include=’ parameter is set correctly in your zabbix agent config file). This will query HAProxy for the “total” connections/sessions (the sum of all seperate underlying hosts).

UserParameter=haproxy.scur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,FRONTEND' | sed 's/,/\ /g' | awk '{print $3}'

And place the following in seperate /etc/zabbix/zabbix_agent.d/haproxy_smtp1.conf files, for each seperate server you want to monitor via HAProxy.

UserParameter=haproxy.smtp1.qcur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $3}'
UserParameter=haproxy.smtp1.qlimit, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $26}'
UserParameter=haproxy.smtp1.scur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $5}'
UserParameter=haproxy.smtp1.econ, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $14}'

Also check the Zabbix Documentation, to allow the Zabbix user to run sudo (/usr/bin/sudo) on the socat binary (/usr/local/bin/socat) without a password prompt: Access Permissions via Sudo (3 – Remote Commands).

The examples above should allow you to monitor your HAProxy appliance in full detail. Good luck!



Want to subscribe to the cron.weekly newsletter?

I write a weekly-ish newsletter on Linux, open source & webdevelopment called cron.weekly.

It features the latest news, guides & tutorials and new open source projects. You can sign up via email below.

No spam. Just some good, practical Linux & open source content.