If you use HAProxy as a load balancing tool, and Zabbix as your monitoring tool of choice (which you should use both, really – I love ‘m), you can use the following template/guide to do some advanced logging of HAProxy’s usage.
It’s based on the same techniques used by hatop to use a “top”-like interface to check your HAProxy setup.
Enable statistics in HAProxy
For starters, we’ll need to modify HAProxy ‘s config, to include the “stats socket” parameter. Add this to the “global” section of the HAProxy config file.
global # Create a socket that allows us to query it directly stats socket /tmp/haproxy.sock
Install ‘socat’ for querying the socket
As discussed in the HAProxy documentation (see section “9.2. Unix Socket commands"), we can query this socket directly and ask it for usage statistics. To do so, use a tool like socat.
$ cd /usr/local/src $ wget http://www.dest-unreach.org/socat/download/socat-1.7.1.3.tar.gz $ tar xzf socat-1.7.1.3.tar.gz $ cd socat-1.7.1.3/ $ ./configure $ make $ make install
Here’s how you can use it.
$ echo "help" | socat /tmp/haproxy.sock stdio clear counters : clear max statistics counters (add 'all' for all counters) help : this message prompt : toggle interactive mode with prompt quit : disconnect show info : report information about the running process show stat : report counters for each proxy and server show errors : report last request and response errors for each proxy show sess [id] : report the list of current sessions or dump this session get weight : report a server's current weight set weight : change a server's weight set timeout : change a timeout setting disable server : set a server in maintenance mode enable server : re-enable a server that was previously in maintenance mode
Querying useful statistics from HAProxy
And some example output of various (interesting) commands: “show info”
$ echo "show info" | socat /tmp/haproxy.sock stdio Name: HAProxy Version: 1.4.8 Release_date: 2010/06/16 Nbproc: 1 Process_num: 1 Pid: 3940 Uptime: 0d 0h14m11s Uptime_sec: 851 Memmax_MB: 0 Ulimit-n: 4112 Maxsock: 4112 Maxconn: 2048 Maxpipes: 0 CurrConns: 3 PipesUsed: 0 PipesFree: 0 Tasks: 6 Run_queue: 1 node: lb02.lab.mojah.be description:
“show sess”: this will dump all known sessions. The following example only has 2 active sessions. Using this on high-traffic systems will be terribly slow, as it’ll show _all_ active sessions. I don’t recommend using it, unless absolutely necessary.
$ echo "show sess" | socat /tmp/haproxy.sock stdio 0x811c820: proto=unix_stream ts=09 age=0s calls=2 rq[f=d0d220h,l=0,an=00h,rx=,wx=,ax=] rp[f=009000h,l=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[7,0h,fd=-1,ex=] exp=10s
“show stat”: will show all statistics in CSV format.
$ echo "show stat" | socat /tmp/haproxy.sock stdio # pxname, svname, qcur, qmax, scur, smax, slim, stot, bin, bout, dreq, dresp, ereq, econ, eresp, wretr, wredis, status, weight, act, bck, chkfail, chkdown, lastchg, downtime,qlimit, pid, iid, sid, throttle, lbtot, tracked, type, rate,rate_lim, rate_max, check_status, check_code, check_duration, hrsp_1xx, hrsp_2xx, hrsp_3xx, hrsp_4xx, hrsp_5xx, hrsp_other, hanafail, req_rate, req_rate_max, req_tot, cli_abrt, srv_abrt, stats,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,, stats,BACKEND,0,0,0,0,2000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,1023,0,,1,1,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0, smtp,FRONTEND,,,0,2,2000,3,0,82,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,, smtp,smtp1,0,0,0,1,,1,0,44,,0,,0,0,0,0,UP,1,1,0,0,1,228,793,,1,2,1,,1,,2,0,,1,L4OK,,0,,,,,,,0,,,,1,0, smtp,smtp2,0,0,0,1,,1,0,38,,0,,0,0,0,0,UP,1,1,0,0,1,230,791,,1,2,2,,1,,2,0,,1,L4OK,,0,,,,,,,0,,,,1,0, smtp,smtp3,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1020,1020,,1,2,3,,0,,2,0,,0,L4TOUT,,2000,,,,,,,0,,,,0,0, smtp,BACKEND,0,0,0,2,2000,3,0,82,0,0,,1,0,0,0,UP,2,2,0,,1,230,790,,1,2,0,,2,,1,0,,1,,,,,,,,,,,,,,2,0,
For instance, in my setup I have 3 listening SMTP servers (smtp1, smtp2 and smpt3 above) in the frontend. I can query the socket, to tell me how many current sessions I have. I can filter for totals “smtp,FRONTEND” or per server “smtp,smtp1”.
$ echo "show stat" | socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $5}'
1
I’m piping the output of my socket-query to grep, to only match the server I want, and I convert the CSV output to a space-seperated output via sed, so I can more easily extract the column I want via awk. If you change the “print $5” line, you can choose to print other columns as well. Here are the more interesting ones.
print $3: current queued requests (qcur) print $4: max queued requests (qmax) print $26: queue limited (qlimit) print $5: current sessions (scur) print $6: max sessions (smax) print $14: connection errors (econ) print $37: status of last health check (should be L4OK, L6OK, L6OK (check_status)
Do note that if you’re grepping for the FRONTEND or BACKEND groups, there will be no output for qcur and qmax, which means all numbers will shift 2 places to left. Meaning scur is not $5, but $3 – as awk won’t count that as an occurence.
Custom UserParameters in Zabbix for HAProxy stats
You can use the following “template” on the HAProxy server to use Zabbix Agent checks and retrieve this data. Be sure to modify the file, so it greps for your correct servers!
Place this file in /etc/zabbix/zabbix_agent.d/haproxy_totals.conf (and make sure the ‘Include=’ parameter is set correctly in your zabbix agent config file). This will query HAProxy for the “total” connections/sessions (the sum of all seperate underlying hosts).
UserParameter=haproxy.scur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,FRONTEND' | sed 's/,/\ /g' | awk '{print $3}'
And place the following in seperate /etc/zabbix/zabbix_agent.d/haproxy_smtp1.conf files, for each seperate server you want to monitor via HAProxy.
UserParameter=haproxy.smtp1.qcur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $3}'
UserParameter=haproxy.smtp1.qlimit, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $26}'
UserParameter=haproxy.smtp1.scur, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $5}'
UserParameter=haproxy.smtp1.econ, echo "show stat" | /usr/bin/sudo /usr/local/bin/socat /tmp/haproxy.sock stdio | grep -i 'smtp,smtp1' | sed 's/,/\ /g' | awk '{print $14}'
Also check the Zabbix Documentation, to allow the Zabbix user to run sudo (/usr/bin/sudo) on the socat binary (/usr/local/bin/socat) without a password prompt: Access Permissions via Sudo (3 – Remote Commands).
The examples above should allow you to monitor your HAProxy appliance in full detail. Good luck!