OCSP stapling with HAProxy. From pitfall to euphoria
in Security on Haproxy, Ocsp-stapling, Certificates, Revocation, Letsencrypt, Bash
It’s that time of the year. Christmas holidays. So I thought to myself. Why be naughty this year? This year I’ll be a good boy.
Jææææs I will (as if 
).
I decided to optimize security and a bit of privacy for the websites I’m hosting. Namely:
- The bengtssondd blog. This site.
- Skakmat digte. A website for my poems.
- Retshjaelp. The personal website of the lawyer Kaaveh Piroz.
After having optimized various HTTP security headers for the above websites I wanted to enable OCSP stapling for them as well. Now, this was more of a challenge. I needed to leap over several obstacles.
They where:
- Understanding OCSP stapling
- Configuring HAProxy to use it
- Automatizing the continued use of OCSP stapling for the certificates used by the websites in question
OCSP stapling
OCSP is an acronym for Online Certificate Status Protocol. It is a way for website servers to save on network IO and performance. A way to increase the privacy of your website users. And it makes your website less vulnerable to failing or overloaded CA provider infrastructure.
OCSP gives you these advantages by:
- Taking away the need for end-users to issue calls to the certificate issuer. By offloading certificate revocation calls to the website server
- Resulting in more privacy to the end-user in that their visit to the website is not revealed to the certificate issuer
- Saving performance for the end-user as their browser will not have to make the lookup
- Websites loading a tiny tad quicker for the end-user than without OCSP stapling
- Saving on the number of necessary certificate lookup calls to the certificate issuer. As an OCSP file is saved for each certificate on the server and data from that file is presented to visiting users
- While not actually saving performance for the web-server, it is implemented in a way that is well thought out, in regards to lessening the performance burden
- CA provider infrastructure is a known target for ill intended people. So by caching OCSP responses a website is less vulnerable to DoS attacks on CA’s
Who doesn’t want that?
Configuring HAProxy to use OCSP stapling
HAProxy is the load-balancer fronting the mentioned websites. Let’s configure it.
All HAProxy requires in order to use OCSP stapling is:
- An
.ocspfile. Named as the crt/pem file it is covering. With the sole addition that it should have the.ocspextension- The OCSP file for e.g.
my-certificate.pemhas to be namedmy-certificate.pem.ocsp
- The OCSP file for e.g.
- The
.ocspfile should be valid and in the DER format. To be valid:- it has to indicate a good status
- it has to be a single response for the certificate of the PEM file, and it
- has to be valid at the moment of addition
More here.
Readying HAProxy
The stats socket will be utilized for the automatization of HAProxy. This socket is not enabled by default. You need to do the following:
- Enable the stats socket in the
haproxy.cfgfile
stats socket /haproxysocket/haproxy.sock mode 660 level admin
stats timeout 2m
add the above to the global section in the haproxy.cfg file.
- Reload/restart HAProxy for the above to take effect
N.B. If you run HAProxy via a Docker container you need to do something along the lines of:
docker run.... -v /folder/to/contain/the/socket:/haproxysocket/-
the folder referenced in the
-vspecification above is the same folder you use in the script below. So by referring to /folder/to/contain/the/socket/haproxy.sock in yoursocatcmd you will be “talking” to the socket inside the HAProxy container.
Generating the OCSP file
So how does one generate the OCSP file. You dig around and find someone like Igor Cicimov. He made a nice script for generating the OCSP file and configuring HAProxy with it. See his blog post. However I had to update the script a few places.
This part was not working on my QNAP:
ISSUER_NAME=$(echo ${ISSUER_URI##*/}I had to change it toISSUER_NAME=$(echo ${ISSUER_URI##*//}.
And because HAProxy did not like the issuer certificate to be in the same folder as the actual website certificates (because the issuer certificate do not contain a private key), I had to update
-out ${DIR}/${ISSUER_NAME}.pemon thewgetandopenssl ocspcommands to/tmp/${ISSUER_NAME}_CHOOSE_YOUR_POSTFIX.pem.
I also removed
[[ $? -eq 0 ]] && [[ $(pidof haproxy) ]] && [[ -s ${DIR}/${CERT}.ocsp ]] &&from theset ssl ocsp...line. As I don’t see the use for it and don’t fully grasp what it is trying to do. Feel very free to ping me if you are able to decipher this black magic. Find my Twitter under “About” in the lower left corner of this site.
After the above changes the script works. However, if you have several certificates and try to run the script for another certificate, the set ssl ocsp..... command will fail, if there isn’t already an OCSP file for the respective certificate.
The error will be: OCSP single response: Certificate ID does not match any certificate or issuer. To alleviate this you’ll have to generate the OCSP file for all your certificates and reload HAProxy.
The above issue apparently happens because of the way that OCSP support for HAProxy was designed. Its initial purpose was to update the pre-loaded OCSP record. So if there is no OCSP file for a certificate already, and you execute the set ssl ocsp..... command, the mentioned error will be thrown.
More on that error here.
The leftovers
With the OCSP file generated and the ability to reload HAProxy when the OCSP file has been updated all there is left now is to do this automatically.
Simply setup a cron job. Timing it to once a day should suffice.
When your are satisfied with your setup and want to see the fruits of your effort. Go-to SSL Decoder and give it a spin. Under OCSP Stapling, in the result table it should say Cert status good and so forth.
Congrats on using OCSP stapling. We can all be happy with that.
Useful resources
- The set ssl ocsp-response cmd
- Talking to the HAProxy socket inside a Docker container
- HAProxy configuration via the API
Thank you for reading along.
Over and out 