HAProxy watch status when no web browser or syslog!
Published: 2020-07-20 15:20:07

---
title: HAProxy watch status when no web browser or syslog!
published: true
description: Tricks when you want to interact with a running  HAProxy load balancer, such as to monitor its selection of backends, debug it, etc, but you can't use web browser, curl or syslog!
tags: 
//cover_image: https://direct_url_to_image.jpg
---
The other day I got a challenge: I had to 1) do a basic setup of haproxy so one frontend would round-robin on 3 backends (simple task), but 2) verify that it was actually round-robin'ing, without using a web browser (to access the stats page that haproxy can be 
configured to serve, eg see “HAProxy Stats Page” on https://www.haproxy.com/blog/introduction-to-haproxy-logging/) or syslog. 

I tried curl which technically is not a web browser, but html was horribly formatted and didn't seem to have the backend status up/down etc; and converting to markdown and plain text with pandoc did not help. 

This won't be news to those who work with haproxy all the time, but for those starting: 

Turns out the haproxy makes status available through a unix socket that can be configured in the `global` section of `/etc/haproxy/haproxy.cfg`. On my centOS it defaults to  `/var/lib/haproxy/stats`, whereas on ubuntu it defaults to `/run/haproxy/admin.sock`. 

After a bit of research I found it was sufficient to send the `show stat` command to the socket, and then process the output. Even better, 
```
watch 'echo "show stat" | sudo nc -U /var/lib/haproxy/stats | cut -d "," -f 1,2,8-10,18 | column -s, -t'
```
on the load balancer host (LB_HOST) to get the stats. Then in a separate terminal (same or different host), I ran
```
while true; do curl LB_HOST:HAPROXY_PORT; sleep 1; done 
```
to generate http requests on the load balancer, and in a third terminal I could add/remove backend servers and observe the load-balancer status up/down. Example output of the watched command:
```
$ echo "show stat" | sudo nc -U /run/haproxy/admin.sock | cut -d "," -f 1,2,8-10,18 | column -s, -t
# pxname       svname    stot  bin    bout   status
test_frontend  FRONTEND  275   21725  67755  OPEN
test_backend   server1   158   12482  39026  UP
test_backend   server2   93    6399   19871  DOWN
test_backend   server3   39    2844   8858   DOWN
test_backend   BACKEND   275   21725  67755  UP
```
Pretty cool!

You can even have a "prompt" where you can do various things other than stats, like take down one of the backend servers for maintenance, debug certificates/SSL, etc. 

Eg with only servers 1 and 2 up (ie 3 is down), I disabled 1 and 2 in haproxy for maintenance, by doing this: 
```
$ sudo nc -U /run/haproxy/admin.sock
prompt
> disable server test_backend/server1
> disable server test_backend/server2
```
then the watch automatically updated to show this: 
```
# pxname       svname    stot  bin    bout    status
test_frontend  FRONTEND  674   53246  164642  OPEN
test_backend   server1   472   37288  116584  MAINT
test_backend   server2   148   11692  36556   MAINT
test_backend   server3   0     0      0       DOWN
test_backend   BACKEND   674   53246  164642  DOWN
```
Notice now the whole BACKEND is down too, because of server 3. Start server 3 and observe its state going from DOWN to UP, and then that of BACKEND does same. 

Enjoy!

A couple of useful refs: 
- [A post on stackexchange](https://superuser.com/questions/1193917/how-to-view-haproxy-status-on-the-command-line-using-a-socket)
- [Section 9 "Statistics and Monitoring" of the docs](http://cbonte.github.io/haproxy-dconv/2.2/management.html#9)