Since the software is not getting better, Varnish it:

home | blog | Terrible people and places | Covid-19 links | Teh Internet | guest blog |rants | placeholder | political | projects | Gwen and Liam | Citadel patched | Tools | Scouts

Just a spot to remember Varnish tidbits.

See what is happening for requests (right now) via varnish:
varnishtop -i rxurl

or for everything in ncsa / apache log format:


If you want counts of who from where (assuming proxy setup):

varnishtop -i TxHeader -I '^X-Forwarded-For:'

Or if you want response time histogram:

Reload the vcl (and not the cache):
service varnish reload

Remember to put the shared memory file on tmpfs:
echo "tmpfs /var/lib/varnish tmpfs nosuid,noatime,nodiratime,size=150M 0 0" >> /etc/fstab'

Default is 80 MB, don't think there is overhead, but keeping it at 150MB should be good for now if we can afford it.

varnishadmin for tuning on the fly. Save to the vcl to make it stick.

To verify Varnish is good to go, make sure to monitor uptime.
Varnish will syslog when the child dies, but the uptime parameter via a local check for Nagios (or your monitoring too of choice) is a better bet.
Since uptime is an up counter, you will need to think about the thresholds for critical and warning.

Logging the x-forwarded-for header
Will Jackson has a great article here:
And as always, in case that link or site goes away, here are the tidbits:
Add the x-forwarded-for block to the varnish config vcl:
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;

Create a new custom log type:

echo 'LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined' > /etc/apache2/conf.d/varnish-log

Change the CustomLog line in the sites you want to see the forwarded ip header from varnish to be the new varnishcombined log format.

For Apache 2.4:


RemoteIPHeader X-Forwarded-For

Remote IP Internal Proxy has less restrictions - i.e. will allow ipv4 RFC 1918 addresses - but feel free to use the more restrictive
RemoteIPTrustedProxy directive as needed.  Use ngrep or similar to see what is being sent in the field for addresses to verify your needs.

Edit /etc/apache2/apache2.conf logging (replaced %h with %a) - old line is one with #
#LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

a2enmod remoteip
a2enconf remoteip
apachectl configtest ( look for ok of course before proceeding with a restart )
systemctl restart apache2.service

Get the current running varnish config:
varnishadm vcl.list
(list of configs).... blah
varnishadm "blah"

If you are getting 503 errors, you can try to capture logs with this to try to debug:
varnishlog -q 'RespStatus == 503' -g request

Probably block the bad actor with iptables (or firewall-cmd rules) - if you use UFW, you are on your own:

iptables -I INPUT -s off.en.der.ip -j DROP

firewall-cmd --add-rich-rule='rule family="ipv4" source address="off.en.der.ip" reject'