Bengtsson Driven Development

When a health check makes its host unhealthy

2020-04-13T00:00:00+02:00

Leading up to this post, I’ve been fiddling with my HushDNS project for quite some time. In the words of the project. It’s all about:

....The DNS queries your system/browser and whatever software you use execute, leave you exposed to tracking by
prying outsiders. Leaving you with little privacy in regards to e.g. which websites you visit. This is so. 
Because the data (requests and responses) involved when using DNS, is in the clear.
Meaning --> everybody can determine the websites and other Internet related services you use.
Furthermore, a DNS query you execute can be mingled with. So you could end up on a website
controlled be ill intended people. Rather on the one YOU intended.

Luckily, it's possible to seriously combat the issues that DNS have......

Early April 2020 I finally got everything to work. From a functionality point of view. However, after a couple of days, DNS queries came to a halt. Either taking a very long time to complete, or not being answered at all. Worse was it, that the host on which dnscrypt-proxy and Pi-hole is running, came to a halt. So I put on my superman debug cape and pursued the issue.

Come with me. Follow my debug tracks to see what I found.

Initial brute-force debugging

So yes, I must admit, the first action I took was restarting the box hosting the HushDNS containers. It did help. However, only temporarily. Not really a surprise though. Excuse me my laziness. Just how it is at times :innocent:.

I quickly came to the conclusion that restarting the host was not going to be a permanent solution. I had to go deeper, way deeper.

Spearhead debugging

The next time it happened, I was lucky enough to already be SSH connected to the container host. So I did my cmd fu. Calling upon top, ps and the like. To my frustration the commands :collision: right in my face. With a very worrying fault: fork: Resource temporarily unavailable. This usually means that a resource limit on Linux has been reached.

As I couldn’t really interact with the host because of its rotten state. I had to restart it again. Hopefully for the last time (related to this issue :wink:) :pray:.

With the host up again. I SSH’ed on to it and decided to do some monitoring to determine what could be hogging resources in such an aggressive way. Furthermore, I went over what recent changes I had made to the host. They where:

Updating the OS of my QNAP
Setting up HushDNS related containers
- Pi-hole had been running for a long time already
- the dnscrypt-server container is running on a host outside my LAN
- dnscrypt-proxy was rather new and had its configuration changed several times

It could be some bug in the newest QNAP OS release. I categorized that as the less likely of the two. I therefore went with looking at the dnscrypt-proxy container. But how is that container resource exhausting the host?

Determining resource limit values

On Linux we do that with the ulimit cmd. E.g. execute:

ulimit -a to eye-gaze values for all the different types of resource limits that can be configured
ulimit -u to determine the value of the max user processes setting

As the fork: Resource temporarily unavailable error is often an issue of too many open files by an user or a max number of running processes met for a user, I had a look at the values of the max user processes- and the open files setting.

max user processes is at 31335 and open files at 1024.

The brute-force way of fixing this would be to set these two to unlimited. But, that would be a peasant solution and would open us up to a fork bomb attack.

Determining the cause of the resource exhaustion

Next up was to determine the current value of max user processes and open files. What to do? Hmmmm let’s pull out the top cmd again and fire it of on the host.

Don’t tell me that you would honestly have any better idea at this point :punch: :stuck_out_tongue_winking_eye:. top is as warm and fuzzy as they come. So let’s hold on to it and squeeze it tightly :older_man:.

Sooooo what did top tell us? :point_down:

.....
   PID PPID User     Stat                     Command
   67     1 root     Z        0   0%   0   0% [drill]
   60     1 root     Z        0   0%   2   0% [drill]
   74     1 root     Z        0   0%   3   0% [drill]
   89     1 root     Z        0   0%   3   0% [drill]
   97     1 root     Z        0   0%   1   0% [drill]
  104     1 root     Z        0   0%   3   0% [drill]
  110     1 root     Z        0   0%   3   0% [drill]
  117     1 root     Z        0   0%   3   0% [drill]
   47     1 root     Z        0   0%   3   0% [drill]
  131     1 root     Z        0   0%   2   0% [drill]
  138     1 root     Z        0   0%   2   0% [drill]
  145     1 root     Z        0   0%   0   0% [drill]
  152     1 root     Z        0   0%   1   0% [drill]
  159     1 root     Z        0   0%   0   0% [drill]
  166     1 root     Z        0   0%   0   0% [drill]
  173     1 root     Z        0   0%   2   0% [drill]
  180     1 root     Z        0   0%   2   0% [drill]
  189     1 root     Z        0   0%   3   0% [drill]
   32     1 root     Z        0   0%   1   0% [drill]
  203     1 root     Z        0   0%   2   0% [drill]
  209     1 root     Z        0   0%   1   0% [drill]
  216     1 root     Z        0   0%   0   0% [drill]
  223     1 root     Z        0   0%   1   0% [drill]
  230     1 root     Z        0   0%   0   0% [drill]
  237     1 root     Z        0   0%   2   0% [drill]
  244     1 root     Z        0   0%   0   0% [drill]
  251     1 root     Z        0   0%   3   0% [drill]
  258     1 root     Z        0   0%   0   0% [drill]
  265     1 root     Z        0   0%   1   0% [drill]
  272     1 root     Z        0   0%   1   0% [drill]
  279     1 root     Z        0   0%   2   0% [drill]
  286     1 root     Z        0   0%   0   0% [drill]
  292     1 root     Z        0   0%   0   0% [drill]
  299     1 root     Z        0   0%   2   0% [drill]
   40     1 root     Z        0   0%   3   0% [drill]
  313     1 root     Z        0   0%   1   0% [drill]
  321     1 root     Z        0   0%   3   0% [drill]
  328     1 root     Z        0   0%   1   0% [drill]
  335     1 root     Z        0   0%   1   0% [drill]
  342     1 root     Z        0   0%   3   0% [drill]
  349     1 root     Z        0   0%   0   0% [drill]
.....

N.B. The above output is a snippet of the output from top. And for good measure I changed the username in the output to the generic value of root

Oh wow! Holy hamster :hamster:. That looks bad bad not good! Something is firing of the drill command again and again for some reason. We can live with that, if drill is actually killed. But its not and therefore just keeps hanging. This is likely to be the culprit of the resource exhaustion state that the hosts ends up in.

Let’s go after the many hanging drill commands and save looking at this potentially being an open files limit reached issue for later.

If we take a look at the output of the top command again, focussing specifically on the stat column we see that it has a value of Z for all of the drill commands. This stands for zombie and in essence means that the process is dead, it has done its job and what’s left is the structure describing it on the Linux kernel. It is in zombie mode because it awaits its parent process to pick up its exit code. Based on that we can conclude that the exit code of the drill command is not picked up. Or it is, but the process is for whatever reason not killed. Duly note that a zombie process have released its recourses. However, it still remains in the system process table and therefore counts against the max user processes limit.

So what is invoking all those drill commands? Let’s use one of my other favorite old timer Linux commands. Namely ps.

Execute: ps -efl -o pid,ppid,user,comm | grep drill

Here is the output (a sample):

.....
 5453 24102 root    drill
 7012 24102 root    drill
16851 24102 root    drill
24217 24102 root    drill
.....

With the -o parameter we defined the columns that the ps command should output. The column to take note of is the second column. Namely the ppid column. ppid stands for parent id. So there you have it, the pid of the process that invoked the drill command.

Next step. Gathering info on the parent process. We simply do that by executing: cat /proc/{pid}/cmdline, in this case replace {pid} with 24102. The result: dnscrypt-proxy-config/config/dnscrypt-proxy.toml. For a reason unknown to me, all whitespace have been stripped from the output. I guess its because the cat command cannot read the special separator character of the content in the process cmdline file. Because, if we execute: vi /proc/{pid}/cmdline we see that the content of the cmdline file is dnscrypt-proxy^@-config^@/config/dnscrypt-proxy.toml^@. So whitespace is marked with a ^@ character sequence. With this knowledge obtained we can conclude that the actual cmdline of the parent process is: dnscrypt-proxy -config /config/dnscrypt-proxy.toml. So :bow: …. there you have it. It’s the dnscrypt-proxy that invokes all these drill commands.

Solving the cause of the resource exhaustion

dnscrypt-proxy runs in a container on my QNAP. In order to determine what is calling the drill command I had a look at its dockerfile. At the time of writing the dockerfile looked like this:

FROM alpine:edge

ENV DNSCRYPT_LISTEN_PORT=5053 \
    DNSCRYPT_SERVER_NAMES="" \
    PUID=1000 \
    PGID=100

COPY entrypoint.sh /

RUN apk add --no-cache dnscrypt-proxy drill \
    && chmod +x /entrypoint.sh

HEALTHCHECK --interval=5s --timeout=3s --start-period=10s \
    CMD drill -p $DNSCRYPT_LISTEN_PORT cloudflare.com @127.0.0.1 || exit 1

ENTRYPOINT ["/entrypoint.sh"]

Here’s a link to the dnscrypt-proxy dockerfile. As it looked at the time of writing.

By eye-gazing the file we can determine that the drill command is used as a part of the Docker container HEALTHCHECK. Oh Shoot! It is actually the HEALTHCHECK making the host “sick”. This is sooooo :point_down:

But why is this happening. Again looking at the dockerfile we can see that the drill cmd queries a DNS server on port 5053. Okay fine. But wait, I have this volume mapping in my dnscrypt-proxy docker-compose YML file.

    volumes:
      - "./conf/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml"

I had a look in the dnscrypt-proxy.toml file and found that I had configured the listening port of dnscrypt-proxy to a different port than the one that the HEALTHCHECK queries. dnscrypt-proxy still works even though there is a mismatch between the HEALTHCHECK and the configured listening port in the dnscrypt-proxy.toml file. But the drill command will of course fail and fail and fail and fail ….. But why isn’t the HEALTHCHECK just returning the || exit 1 part? And the drill command killed!

Let’s go into that.

For King Canute :crown: let’s peep at that dockerfile once more. Notice that --timeout=3s? Yes you do, you see it :eyes:. After spotting that, I suspected this being an issue of the Docker HEALTHCHECK not leaving enough time for the drill cmd to timeout in the case that the DNS server is not answering. Reading the documentation on the Docker HEALTHCHECK reveals that if the cmd used in the HEALTHCHECK takes more than, in this case, three seconds, the HEALTHCHECK will be considered as unhealthy. More here. After executing the cmd used in the HEALTHCHECK inside the dnscrypt-proxy container, I could conclude that the three second timeout would certainly not be enough to allow the drill command to naturally time out.

I used these commands:

time drill -p 5354 cloudflare.com @127.0.0.1 < query on a port that the dnscrypt-proxy is not listening on
- Result: 15.01s
time drill -p 5053 cloudflare.co @127.0.0.1 < query on the port that the dnscrypt-proxy is listening on
- Result: 0.00s. The speed of the answer is because it was served from the DNS cache

Conclusion

When the Docker HEALTHCHECK times out after three seconds the exit code of the drill cmd is not caught and it just hangs as a zombie process and counts against the max user processes. Then when the max is reached. The disaster occurs as the host experiences resource exhaustion.

The fix

Initially I fixed this by simply making sure that the listening port configured in the dnscrypt-proxy.toml file matched the one the Docker HEALTHCHECK uses. However, after running the container for a couple of days I still saw zombie drill cmd’s. Though a very limited number, compared to before. But of course the wise one thinks. E.g. if for some reason the drill command cannot reach the local DNS server. If the domain does not exist or an network issue occurs, drill does not get the necessary time it needs to time out naturally. Leaving us with hanging zombie drill processes.

The real fix

As part of my troubleshooting I created this GitHub issue. On the repository for the dnscrypt-proxy container image I use. For better or worse I was referred to the klutchell dnscrypt-proxy container image by the image creator. As the creator thinks that image is better maintained. I therefore had a look at that projects dockerfile and what do I see. No HEALTHCHECK. Hmmmm, interesting. Further investigation. This time into the commit history of the klutchell image. In there I found a commit with this revealing title: remove drill and internal healthcheck. But the commit had no reasoning on why. So I created an GitHub issue on the klutchell image repository. Where I simply asked for the reason behind removing the HEALTHCHECK. The creator told me that he did so in order to simplify the image and give users control on whether or not they want a HEALTHCHECK. That knowledge made me decide to continue with the image I was using. However, with the change that I would control its HEALTHCHECK in the docker-compose file I’m using for the dnscrypt-proxy container.

Like so:

version: '3.4'
....
....
....
    healthcheck:
      test: ["CMD", "drill", "-p", "5053", "cloudflare.com", "@127.0.0.1", "||", "exit", "1"]
      interval: 20s
      timeout: 20s
      retries: 3
      start_period: 5s
....

N.B. the above is a snippet of the docker-compose file. Ensure to specify the docker-compose file version to be v3.4 or above, if you want use the start_period parameter.

Overwriting the HEALTHCHECK gives the following benefits:

Controlling the value of the timeout HEALTHCHECK parameter (the actual fix)
And, if I want to, the freedom to set the dnscrypt-proxy listening port as I see fit

Completing that, I then manually monitored the dnscrypt-proxy container for the next day or so. It ran without any issues and even better, there was no longer any hanging zombie processes. There you have it, job done :dart:.

Thank you for reading along. I hope you enjoyed it and that you found useful knowledge.

Over and out :dash: and stay :house:

HushDNS, can I please get me some DNS privacy

2020-04-02T00:00:00+02:00

Motivation & background

I continuously try to improve my digital privacy. Which in todays world also means my general privacy.

So DNS.

Hmmm I don’t really need to say much more. But, I should. To properly set the scene and give an overview of the current state of domain name to IP issues.

The DNS queries your system/browser and whatever software you use execute, leave you exposed to tracking by prying outsiders. Leaving you with little privacy in regards to e.g. which websites you visit. This is so. Because the data (requests and responses) involved when using DNS, is in the clear. Meaning –> everybody can determine the websites and other Internet related services you use. Furthermore, a DNS query you execute can be mingled with. So you could end up on a website controlled be ill intended people. Rather on the one you intended.

Luckily, it’s possible to seriously combat the issues that DNS have. The rest of this post will be about how-to do exactly that.

Let’s do this!

Motivation & background

Action points

Make DNS queries anonymous
- Avoid being logged in as many locations en route to a DNS authoritative server answering ‘x’ DNS query
Encrypt DNS communication. From today until Halloween … so always ultra
Use some form of response validation control. So that the response can be controlled for integrity and forging (e.g. DNS cache poisoning)
The origin of ‘x’ DNS query should be as hard to track back to as possible
Give this project a name. Rememberable and catchy. All IT projects need such a name. Do they not?

Determining resolutions to the action points

Charged with motivation I then started investigating, analyzing and trawling through information on how-to make DNS private and secure. After scouring the corners of the Internet and reading plenty of blogposts and whatnot. I felt ready to get into the heat of the action.

A boring life is a life without action

Anonymity and logging

To avoid being logged and be in greater control of potentially what my DNS queries leak. I came to the conclusion that I needed to run my own DNS recursive name-server.

I ended up choosing Unbound. It is OSS. I can run it on my own hardware. And it can be containerized. The documentation on it is solid and it is build be a highly skilled group of people at NLNETLABS.

Because Unbound have support for DNS Query Name Minimisation. Your privacy is improved by only sending the parts of a domain name, that the different actors involved in a DNS query, really needs to process a request. That alleviates the pain of leaking DNS query data. Good stuff :+1:.

N.B. specifically I’ll be running Unbound as part of the dnscrypt-server in a container “package”.

Encrypt all the things

But with what? I’ll tell you my DNS security compadre.

With….

dnscrypt-server

In the words of the dnscrypt-server project, it is:

- A minimal Ubuntu Linux as a base image.
- Caching resolver: Unbound, with DNSSEC, prefetching, and no logs. The number of threads and memory usage are automatically adjusted. Latest stable version, compiled from source. qname minimisation is enabled.
- encrypted-dns-server. Compiled from source.

Keys and certificates are automatically rotated every 8 hour.

This is great. Some really cool people made this project. Ripe and ready for me to piggy back on. I was quite ecstatic when I found it.

It fits right into the needs of the HushDNS project. Because:

DNS queries will be encrypted
It can easily be run on an external server. Making it harder to traceback the origin of the DNS query
especially combined with the Anonymized DNS feature of dnscrypt-server.

dnscrypt-proxy

It wraps a DNS recursive name-server that natively does not support encrypting DNS queries. Enabling communication from the DNS recursive name-server to the externally running dnscrypt-server. To be wrapped in this case is le Pi-hole. Oui oui. More on Pi-hole in a few.

Both of the above systems is developed by the DNSCrypt project. Thank you for that! Great stuff :smile:.

They are OSS and have solid documentation. It’s all good in the hood-ie.

Integrity and anti-forging

The answer is DNSSEC (Domain Name System Security Extensions) provides anti-forging and validation.

Limit tracking

I’ve already been onto this subject. As I mentioned in the Anonymity and logging section. Unbound gives me control over DNS query logging as well as the DNS Query Name Minimisation feature. But there is more to combatting being tracked.

When you visit e.g. a website a request for a name to IP translation takes place. As DNS is designed, this request will be in the clear. If we encrypt it, use DNS Query Name Minimisation, avoid it being logged in as many places as possible and use DNSSEC to get integrity and prevent forging. We traverse into greener fields.

But <- there it was - Hi!

When a domain name has been properly translated to an IP address that IP address will be accessed. Of course. It’s your ticket to the destination website. If this is done over a tunneled connection (HTTPS) the SNI HTTP header will be sent along with the request. So that the receiving end (the web-server) can serve the correct certificate (public key cryptography my friend). This header reveals what you are visiting. If HTTPS is not involved, the web-server will look at the hostname part of the URI. However, everything is in the clear anyway. So tracking in that scenario is easy peasy.

Hey! Mr. Lars Bengtsson. Haven’t you heard. ESNI (Encrypted Server Name Indication) is a thing now! Yes I have. But it is not supported by all web-browsers. And if you rely on e.g. the Firefox <-> Cloudflare DoH ESNI solution. You basically trust them not to forge and log your DNS requests. This is an issue. By some, considered to be of major proportions. See:

DNS Privacy at IETF 104

More DOH

Both articles are by Geoff Huston. The chief scientist at APNIC (the Asia Pacific Network Information Centre)

However, I’ll try to see if I can configure Firefox to use my DoH DNS setup. Then again, I control logging or no logging. Encryption and on we go. Follow along to see if I succeed. If (when) I do, ESNI can be used and we are a tad more safe. However, there is still the caveat that the web-server hosting the destination website have to support ESNI. So we are not home-free. And even with ESNI there is PLF (Page Load Fingerprints), which is a way to determine the website you are accessing by analyzing the meta-data “surrounding” that visit. In other words. Even though your DNS query was secure, unforgeable and so forth. Accessing the IP address that was returned as a response to the DNS query and the other resources involved in loading a website, can reveal the identity of the website you are visiting. Because most websites have a unique set of resources they need. Integration with facebook (need, yeah right), ad networks, CDN’s and on we go. So basically. Identify this set and you know the website being visited.

It is possible to combat PLF. By using an ad-blocker that works on the application layer. By blocking DNS requests to the resources that ‘x’ website thinks it needs, but doesn’t really, for the website to be useful to you. This is where Pi-hole comes in.

E.g. you can often block ads or metrics software on a website and the website continues to work. In this way, prying eyes, trying to identify the website you are visiting, would not be able to utilize PLF. As the resources actually being requested when accessing a website do not match the PLF set determined by analyzing the website.

But the issue around SNI and ESNI still persists. Nothing else to do there than push for companies to host their website on platforms supporting AND using ESNI. Unless you go the VPN or Tor route. However, using Tor comes with its own caveats. See this.

Executing on the findings

A jaw dropping project name

HushDNS it is. Because those DNS queries should “shut up and be quiet”. I’ll let it be up to you to grade the name.

I like IT - and I like it a looooot :dash:. HushDNS, who can’t remember that?

N.B. the rest of the practical part of this blog post, will be going through the components one by one. In order to detail how each component work. However, if you quickly want to reap the benefits of using HushDNS follow the installation guidelines on the HushDNS repository

Resolving the anonymity and logging issue

Blocking DNS requests at the Application layer

Pi-hole is a blackhole for Internet advertisements. Technically you configure it to be your DNS server on your LAN and then, when ‘x’ software on your computer needs to resolve a domain name to an IP address. Pi-hole is requested to do that. If it finds that the requested domain name is on a blacklist it responds with its own IP, and the software making the request gets an empty payload back when requesting data from the returned Pi-hole IP. This results in an empty space on the website you are accessing. E.g. where an ad should have been shown.

This is a great way to combat the effectiveness of page load fingerprinting. Simultaneously you limit the amount of 3rd parties that know of your IP and website surfing habits.

Configuring Pi-hole

I’ll be running Pi-hole in a container. Using this docker-compose.yml file.

Kick of the Pi-hole container

Download the above mentioned file
Execute: docker-compose -f PATH_TO/docker-compose.yml -p "PiHole" up -d
1. This will install Pi-hole. Name the compose “project” and container PiHole and detach from the container

Settings for the Pi-hole container

The Pi-hole docker-compose file is specified to setup Pi-hole in the following way.

CONDITIONAL_FORWARDING: "true": Used in the case that Pi-hole is not the DHCP server on the network
CONDITIONAL_FORWARDING_IP: "IP_ADDRESS": The IP address of the DHCP server on the network
CONDITIONAL_FORWARDING_DOMAIN: "DOMAIN_NAME": A DNS suffix name. It has to match the DNS suffix name used in the DHCP server
DNS_BOGUS_PRIV: "true": Specifies that reverse IP lookups for LAN IP addresses should never be forwarded to upstream DNS servers
DNSMASQ_LISTENING: "local": Controls which network interface dnsmasq will listen on. Specifying “local” results in > Allows only queries from devices that are at most one hop away (local devices).
DNS1 and DNS2. As Pi-hole complains if you don’t specify two addresses I had to trick it to query on the localhost IP and the regular IP of the same network interface. The port part, #5354 is important. This is the port that dnscrypt-proxy listens to. And that is where queries from Pi-hole is being sent. More on that dnscrypt-proxy later on
ServerIP: "IP_ADDRESS": Needed when running the container in host network mode
TZ: 'Europe/Copenhagen': The timezone that Pi-hole runs in
WEB_PORT: "808": Need when Pi-hole runs in host network mode. Can break that you see a “beautiful”, blocked by Pi-hole block where an ad would’ve been displayed. I can live with that
dns: - 127.0.0.1. Used to specify the DNS server used inside the container. Should end up in the container /etc/resolv.conf file. However, not always do. There is an issue on this on GitHub. Still have the setting here, to remember the issue and that this is the intended configuration
volumes: - "./etc-pihole/:/etc/pihole/" && - "./etc-dnsmasq.d/:/etc/dnsmasq.d/": To store Pihole data between upgrades
volumes: - "./resolv.conf:/etc/resolv.conf:ro": A trick I needed to do. More on this in the QNAP specifics section below.

N.B. I haven’t configured Pi-hole to use DNSSEC as the dnscrypt-server is per default configured to do this. And validating DNS answers via dnssec in several locations will not do us any good. Just slow things down and potentially causes failed DNS queries. However, if you are not going to use the dnscrypt-server make sure to enable it via the DNSSEC environment variable in the docker-compose file

QNAP specifics

If you’ve been following my blog here, you will know that I run much of workloads on a QNAP NAS. Because of that, I often times need to go through loopholes to get “things” to run like I want it. Which in turn is also why I’m often considering abandoning my QNAP.

So here is another one of those sections :unamused:. How-to get ‘x’ to run properly on a QNAP.

Fixing resolve.conf
1. Create a resolve.conf file with the following content. The file needs to be located relative to the resolve.conf container volume mapping mentioned in The settings above

nameserver 127.0.0.1
options ndots:0

this content will overwrite whatever is in the etc/resolv.conf inside the Pi-hole container. Making dns work as intended inside the container.

This setting also relates to the dns: - 127.0.0.1 setting. As it didn’t always work without overwriting the resolve.conf file.

Disable dnsmasq. To do that execute the below

cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
sed 's/port=53/port=0/g' < /etc/dnsmasq.conf.orig >/etc/dnsmasq.conf
/usr/bin/killall dnsmasq

Alternatively follow Running your own application at startup (on QNAP), to avoid doing that on every QNAP boot

Useful docs

Tumbling DNS queries

By using dnscrypt-proxy we gain access to the DNSCrypt projects Anonymized-DNS feature. This feature makes it harder to origin trace our DNS queries. It does its magic by … aaah no, allow me to use a word by word copy of this description. So:

Instead of directly reaching a server, an Anonymized DNS client encrypts the query for the final server, but sends it to a relay.

The relay doesn't know the secret key, and cannot learn anything about the content of the query. It can only blindly forward the query to the actual DNS server, the only server that can decrypt it.

The DNS server itself receives a connection from the relay, not from the actual client. So the only IP address is knows about is the one of the relay, making it impossible to map queries to clients

Considering Anonymized-DNS when you run your own `dnscrypt-server`

The question is, however, if it’s necessary and/or beneficial to use Anonymized-DNS if you run your own dnscrypt-server. In comparison, you really should if you use an external provider like CloudFlare.

After having used Anonymized-DNS for several days, with my own isolated dnscrypt-server I weighted the pros and the cons. Then re-considered. The feature is simply not stable enough. Then, when it also entails added latency, it is currently not worth it. But, I will revisit this in the feature as it does add an extra layer of anonymization.

For good measure. Here is my list of pros and cons:

Pros

Extra anonymity, yes even if you run your own dnscrypt-server, specifically if someone tries to sniff on you. By analyzing traffic between the dnscrypt-server and the dnscrypt-proxy they could be able to see that its DNS traffic going out of the dnscrypt-server endpoint and then on the other side, what IP that’s connecting to it. And if its very few IP’s, it should be possible to analyze the source of the DNS traffic and thereby de-anonymize your :shit:.

Cons

Extra latency
Depending on the external Anonymized-DNS relay servers being up and working
The implementation of Anonymized-DNS in dnscrypt-proxy, at least up until v2.0.42, selects one relay server and then stays on that until reboot. So if that relay goes down you’ll have to reboot. Instead the Anonymized-DNS feature should be able to switch over to another configured relay server if there is problems with the currently selected one
Added complexity when troubleshooting

Configuring dnscrypt-proxy

Another workload I’ll be running in a container. This time by using this docker-compose.yml file.

Kick of the dnscrypt-proxy container

Download the above mentioned file
Execute: docker-compose -f PATH_TO/docker-compose.yml -p "dnscrypt-proxy" up -d
1. This will install dnscrypt-proxy. Name the compose “project” and container dnscrypt-proxy and detach from the container

Settings for the dnscrypt-proxy container

DNSCRYPT_LISTEN_PORT: “5354”: Self-explanatory
DNSCRYPT_SERVER_NAMES: “[‘MY_SECRET_DNSCRYPT-SERVER’]”: The dnscrypt-server or DoH server that dnscrypt-proxy should connect to
network_mode: “host”: The port was not reachable for Pi-hole if this network mode was not used

In order to setup Anonymized-DNS we need to complete the following steps (if you still want to set it up, considering the pros and cons of Anonymized-DNS):

Ensure that there is a sub-folder named conf in the folder of the dnscrypt-proxy docker-compose.yml file 1. In this folder create a file named dnscrypt-proxy.toml
Use this dnscrypt-proxy.toml file as a template
1. Change the value of server_names in the Global settings section to the name you will give your dnscrypt-server
2. Potentially change the listen_addresses to the port you want (in the Global section)
3. Under the Anonymized DNS section change the routes array to contain one or more Anonymized DNS relay servers of your choice. Find available relay servers here. Make sure to change the server_name in the routes definition, to reflect the value of the server_names property in the Global section

N.B. if you look at the dnscrypt-proxy docker-compose.yml file you’ll notice that there is a volumes mapping. This volume mapping is what the above steps relate to

I said: “don’t log me”

And “they” listened. Good for them. So what “they” did was to ensure that neither Pi-hole, dnscrypt-proxy or dnscrypt-server was in any form of logging mode.

Disabling logging in Pi-hole

When you feel certain that your Pi-Hole installation is working, it’s a good time to disable its logging behavior. By:

Going to Settings in the Pi-Hole web-interface
Clicking on the Privacy tab
Then activating the radio button: Anonymous mode: This disables basically everything except the live anonymous statistics…... This setting is very privacy minded but still makes it possible to verify that Pi-Hole is actually blocking certain queries. You’ll also still have the main stats available, now just politely anonymized

There is currently no way to do the above in an automated fashion. As the Pi-hole in a docker container project does not support configuration on this on container initialization. However, you could overwrite the PRIVACYLEVEL=0|1|2|3|4 (optionally read this) setting, by using a Docker volume mapping to /etc/pihole/ in order to overwrite the pihole-FTL.conf file with your own customized one. I chose not to, as this is the only setting I would need to overwrite. Making the extra work required too much in my opinion

Disabling logging in dnscrypt-proxy

Not much to do here. The dnscrypt-proxy is already non-logging in its default setup.

Disabling logging in dnscrypt-server

Not much to do here. The dnscrypt-server is already non-logging in its default setup.

Encrypting all the things

This is achieved by using dnscrypt-proxy and dnscrypt-server.

As I have already explained how-to setup dnscrypt-proxy and there is no more to it. This part focuses on the dnscrypt-server.

Configuring dnscrypt-server

As dnscrypt-server needs an init container, and that is not supported by the Docker engine, you’ll have to do with a shell script.

Download this bash script and ensure to update the placeholders (capital words, separated by an underscore) with actual useful values
1. Change the port of the dnscrypt-server if you need to (already have ‘x’ service running on port 443). Change from port 443 to “your” port, in all the locations where it is specified
2. Make the file executable by executing: sudo chmod +x ./THE_NAME_YOU_GAVE_THE_FILE
Execute the file on the system that is to host the dnscrypt-server
Note down the output of the init -N NAME_TO_GIVE_YOUR_DNSCRYPT_SERVER... command as you need the info when configuring dnscrypt-proxy
1. you can also get the output after the fact by executing docker logs dnscrypt-server
2. The output to copy is the generated stamp. You need this in order to connect to the dnscrypt-server via the dnscrypt-proxy

Using it together with `dnscrypt-proxy`

As we can see in the Settings for the dnscrypt-proxy container section. Configuring the dnscrypt-proxy involves its dnscrypt-proxy.toml file. So, in order to use the dnscrypt-server you just spun up together with your dnscrypt-proxy server, you need to update the dnscrypt-proxy.toml file.

Find the [static] section in the file
Change the server name part of [static.'hush.dns'] to the server name you’ve used throughout the dnscrypt-proxy.toml file
Finally set the value of the stamp property to the DNSCrypt stamp that the dnscrypt-server spit out when it was initialized

Verifying that the dnscrypt-server works

N.B. if you are on Docker Desktop for Mac you cannot ping or use dig/drill directly to the container. Even though it is in host network mode. This is because Docker Desktop for Mac cannot route traffic to containers. That's the reason for the below way to test whether the dnscrypt-server` is working

With a DNS query

On Mac

Execute: docker exec -it DNSCRYPT_PROXY_CONTAINER_NAME /bin/sh
Then: drill -p THE_PORT_YOU_CHOSE @127.0.0.1 SOME_DOMAIN

On Linux

I haven’t tested this but it should be possible to execute: drill -p THE_PORT_YOU_CHOSE @127.0.0.1 SOME_DOMAIN directly to the dnscrypt-proxy container. In other words you don’t have to exec into the container.

For both of the above examples. If you get back a valid answer you know that the dnscrypt-server works. To get even more reassurance, stop the dnscrypt-server container and try again. Now you should get no answer back. Rather a timeout/network error.

Verifying the use of Anonymized-DNS

It makes the most sense to verify Anonymized-DNS from the point of the dnscrypt-server. As this is the server that will get the dnscrypt-proxy forwarded DNS query. And therefore is the server that should NOT see your WAN IP (from where the dnscrypt-proxy is running). Rather the WAN IP it sees making a DNS query is an Anonymized-DNS relay server.

On the server where the dnscrypt-server is running execute: tcpdump -nnSX port 443 this will sniff on HTTPS (port 443).

An example output could look like:

20:35:10.770590 IP SOME_WAN_IP.443 > ANONYMIZED_DNS_RELAY_SERVER_WAN_IP.38756: UDP, length 240
        0x0000:  4500 010c 2a4a 4000 4011 6c80 82e2 d9de  E...*J@.@.l.....
        0x0010:  5410 f245 01bb 9764 00f8 a420 7236 666e  T..E...d....r6fn
        0x0020:  7657 6a38 f984 9650 3145 710b a9ff c436  vWj8...P1Eq....6
        0x0030:  f999 4174 d7ae a874 b313 08f7 aa9b 7c9c  ..At...t......|.
        0x0040:  5dc6 0da6 3c19 1adc d272 30ee 0927 472a  ]...<....r0..'G*
        0x0050:  83e5 6fa9 8ac8 7a9f a8c7 6201 1dbf ff06  ..o...z...b.....
        0x0060:  7703 e04c 0023 c681 6e47 7b37 1b2b cb35  w..L.#..nG{7.+.5
        0x0070:  a740 edbc c232 0fb5 15d5 8036 fe57 6525  .@...2.....6.We%
        0x0080:  5925 e324 d25b 376d 5d0a 4e4d 320d c6d8  Y%.$.[7m].NM2...
        0x0090:  58d2 7697 1aff d2f0 37a7 cc19 48be d066  X.v.....7...H..f
        0x00a0:  1e68 ff34 fb35 d231 651f 8b33 0033 93e1  .h.4.5.1e..3.3..
        0x00b0:  6ceb f9e1 1676 a592 f69e fe62 a3f5 688f  l....v.....b..h.
        0x00c0:  2db6 ea30 0e6b ff56 fa8a 4223 4520 767c  -..0.k.V..B#E.v|
        0x00d0:  79d1 dcb9 cf66 6aba a748 2de8 223f d2dd  y....fj..H-."?..
        0x00e0:  d2af 3179 47ac 85a6 b7cd a8ab ef7a cfa9  ..1yG........z..
        0x00f0:  ee99 9df9 70ce 2049 c657 f686 906c 0c41  ....p..I.W...l.A
        0x0100:  1f13 1f07 dd0c 0154 0871 4a46            .......T.qJF

You can be all :+1: up if the WAN IP making the DNS query is not your WAN IP but the WAN IP of a DNSCrypt Anonymized-DNS relay server. Getting the IP of a relay server is easy with the online DNSCrypt Stamp calculator. Just insert the stamp of the Anonymized-DNS server you use in the stamp field on the page and you should be able to see the WAN IP of the relay server in the IP Address field :clap:.

Optionally you can go-to the DNS leak test site to see whether it’s your private WAN IP or a WAN IP of an Anonymized-DNS relay server.

Don’t forge my integrity

As you may remember, the answer is DNSSEC. We got a little :gift: here. In that it is already enabled when we use the jedisct1/dnscrypt-server container image. The devil is the detail, or maybe in the detail. Whatsoever, DNSSEC is availble via the Unbound DNS recursive name-server that dnscrypt-server uses internally. Therefore gifting us with DNSSEC.

How DNSSEC works and some background

To my surprise the IETF (Internet Engineering Task Force) have been working on DNSSEC since 1993.

It is designed to protect against:

packet interception (in its various forms)
ID Guessing and Query Prediction
Name chaining (a subset of “cache poisoning” attacks)
Betrayal By Trusted Server. E.g. an ISP DNS server configured to forward statistics to a 3rd party unbeknownst to the end-user
Authenticated Denial of Domain Names
Data integrity and data origin authentication for “wildcard” DNS names

However, it does NOT protect against:

modification of the DNS message header. Meaning that the response is digitally signed. But the message packet header could still be mingled with. A properly configured DNS recursive name-server, however, combat that
unless configured to do so, DNSSEC does not cover glue records, which are records that combat circular lookup dependencies, e.g. when ‘x’ domains name-servers are on the same root domain as the general resource records
Denial of Service. Actually enabling DNSSEC might make the problem worse

It works by….nah, let’s just unscrupulously copy in the description on wikipedia

DNSSEC works by digitally signing records for DNS lookup using public-key cryptography. The correct DNSKEY record is authenticated via a chain of trust, starting with a set of verified public keys for the DNS root zone which is the trusted third party. Domain owners generate their own keys, and upload them using their DNS control panel at their domain-name registrar, which in turn pushes the keys via secDNS to the zone operator (e.g., Verisign for .com) who signs and publishes them in DNS.

N.B!

DNSSEC does NOT give us confidentiality. That is why we need a solution like the dnscrypt-server. That wraps the Unbound DNS recursive name-server in the encrypted-dns-server proxy. Only then, prying eyes will not be able to sniff (on) the data-stream between the dnscrypt-proxy and the dnscrypt-server.

Configuring DNSSEC

Both the dnscrypt-proxy container, if you use the example dnscrypt-proxy.toml file, and the dnscrypt-server is all ready and prepped for using DNSSEC. Nothing more to do. Perfect days :gem:

Verifying that DNSSEC works

Enter the dnscrypt-proxy container by executing: docker exec -it dnscrypt-proxy /bin/sh
Fire of: drill -p PORT_DNSCRYPT_PROXY_LISTENS_ON @127.0.0.1 dig sigfail.verteiltesysteme.net
1. Should return ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 64618 if it doesn’t on your first try (you might see Error: error sending query: Could not send or receive, because of network error), try again.
Then fire of: drill -p 5354 @127.0.0.1 dig sigok.verteiltesysteme.net
1. Should return ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 36866 and the A record of the domain in the resulting payload

DNSSEC documentation

Limiting tracking

DNS Query Name Minimisation

Already enabled in the default dnscrypt-server settings (more specifically, its actually the Unbound configuration). So that can be :white_check_mark:

SNI & ESNI, Firefox

I decided that it is too early days for using ESNI. It is simply not supported broadly enough. Still very much an experimental feature and the performance cost is too steep to my liking. The dnscrypt-proxy project describes some facts on ESNI to consider here.

In all fairness I should mention that it would be possible to use ESNI by taking advantage of the dnscrypt-proxy and then configure Firefox to direct its ESNI DoH traffic through your dnscrypt-proxy. Therefore, if you still feel that ESNI is something you want to experiment with, then follow the dnscrypt-proxy ESNI guide

PLF AKA Page Load Fingerprints

I’ve already touched upon this issue in the Limit tracking section.

Here we are at the end of the rainbow … and what do we, you, somebody find?

Nothing more. Sorry for destroying your imagination :smile:.

What you did find via this post though, was a HushDNS setup to make your DNS life more private, anonymous and secure. What a tremendous :gift:.

Over and out :dash:

Signaling without exposing your private phone number

2019-12-28T00:00:00+01:00

The situation

I wanted a way to avoid revealing my private phone number when using Signal. There can be many reasons for this. E.g.

Privacy
Safety
Annoyance over why it is necessary in the first place when using Signal

End goal

Accomplishing the above will be considered a success when:

I’m able to use the Signal Desktop application both with my private phone number and my public phone number. On the same computer
Linking the Signal Desktop application can happen without using a secondary phone or needing to uninstall and install the Signal mobile application (as this would result in a need to be verified by my private contacts)
Getting a public phone number should be as cheap as possible
If possible, avoiding the registration of my address
I should be able to hold onto the public phone number for as long as I want

Findings

Using `Signal Desktop` with both phone numbers on the same computer

The Signal Desktop application is not multi-instance capable in the non-development version. Therefore one need to run the general available version and the beta version side-by-side.

Linking without a secondary phone

This proved to be possible even though it required some nice nerdy magic

As cheap as possible & registration

I spent a lot of hours on different vendor pages. Vendors selling virtual phone numbers and the like. Twilio ended up being the vendor I chose. As with them it is possible to acquire an US phone number for 1$ a month. With an initial top-up of 20$. Therefore giving you 20 months of Twilio.

Twilio also makes it possible to use Paypal as a payment service. Limiting the number of stakeholders that will get parts of my personal information.

Holding onto the phone number

Using a prepaid mobile phone card is not an option. Even though it nicely comes with no registration. As the phone number falls out of my control at some point. Per say, you’re leasing it.

Again Twilio makes this possible. As long as I pay my dues.

Guidance on achieving the above

Parts of the below, have for documentation purposes, been copied from Shields Up: Using Signal Without Giving Your Phone Number

Sign-up for Twilio
1. Verify your account as asked
2. Top-up the account via Paypal with the required 20$
3. Choose an American phone-number, with voice and SMS capabilities
4. In Twilio, navigate to “All Products and Services” > “Runtime” > “TwiML Bins”
5. Create two TwiML bins
  1. One for SMS, that should contain the following code
```
  <?xml version="1.0" encoding="UTF-8"?>
 <Response>
 <Message to="YOUR PHONE NUMBER">
     : 
 </Message>
 </Response>
```
  1. And the one for voice
```
  <?xml version="1.0" encoding="UTF-8"?>
 <Response>
 <Dial>
     YOUR PHONE NUMBER
 </Dial>
 </Response>
```
  N.B. In the above, insert your phone number. When doing so, it should contain the country code it belongs to. Pre-fixed with the plus character.
6. Now navigate to “All Products & Services” > “Phone Numbers” > “Manage Numbers” > “Active Numbers”
  1. Click on your Twilio phone number. Change the “A call comes in” field from “Webbook” to “TwiML” and switch to your custom voice forwarding settings by selecting the appropriate bin name (e.g., “my voice forwarding”). Do the same for the messaging settings.
Install the beta of Signal
1. Windows: First, download this file and look for the url property that specifies the location for the latest beta installer. Download the installer by constructing a final URL that looks like this: https://updates.signal.org/desktop/<installer location>. Then run the installer
2. macOS: First, download this file and look for the url property that specifies the location for the latest beta installer. Download the installer by constructing a final URL that looks like this: https://updates.signal.org/desktop/<package location>. Then unzip that package and copy the .app file into the /Applications folder using Finder
3. Linux: Follow the production instructions to set up the APT repository and run apt install signal-desktop-beta
If it isn’t already install the official, general available version of Signal. As you will have to run the Beta and the Official one side-by-side. A further guide on how-to install the Official one would be the one on Signal’s homepage
Install signal-cli which is a command-line interface for Signal and will be used for registering, linking and testing Signal with the acquired public phone number. Get it from Signal-cli download
You are now ready and prepped for registering Signal with your phone number provided by Twilio. So change directory to the folder whereto you installed signal-cli and…
1. Execute: ./bin/signal-cli -u PHONE_NUMBER_AKA_USERNAME register
2. Execute: ./bin/signal-cli -u PHONE_NUMBER_AKA_USERNAME verify THE_CODE_YOU_RECEIVED_ON_YOUR_PHONE
3. To test that the new Signal account with the Twilio phone number works, execute: ./bin/signal-cli -u PHONE_NUMBER_AKA_USERNAME send -m "crazy stuff" PHONE_NUMBER_AKA_USERNAME_TO_PRIVATE_ACCOUNT
4. Optionally execute: ./bin/signal-cli -u PHONE_NUMBER_AKA_USERNAME_TO_PRIVATE_ACCOUNT receive if you want to send a message the other way. Meaning, from your private phone number to the Signal account with the Twilio phone number
5. Now we want to link the Signal account with the Twilio phone number, but without using a secondary phone
  1. Ensure that Signal Beta is started, you must see the QR code screen in the application
  2. Scan the QR code with a QR scanner on your phone, to derive the URI that it contains
  3. Get the URI to your computer in one way or the other
  4. Now with signal-cli execute: ./bin/signal-cli addDevice -u PHONE_NUMBER_AKA_USERNAME --uri "tsdevice:/?uuid=....
    1. In the Signal Beta application you should now be prompted to name the device
    2. Name it and proceed, Signal Beta will load users and groups and finally …. after some loading your are in :+1:

Optionally heighten the security of your new Signal Twilio phone number account by setting a Registration Lock pin. Here’s how:

Execute: ./bin/signal-cli -u PHONE_NUMBER_AKA_USERNAME setPin PIN_BETWEEN_4_20_CHARS

And make sure to read the Registration Lock article by Whisper Systems. As there is some pointers to be aware of. Namely that the pin expires if there are no activity on the account for more than 7 days.

Readings I found useful

Does Signal send my number to my contacts? read Whisper Systems notes on that here
How to Use Signal Without Giving Out Your Phone Number read The Intercept article on that here
How to install and use Signal messenger without a smartphone read that guide here

Thank you for reading along.

Over and out :dash:

OCSP stapling with HAProxy. From pitfall to euphoria

2018-12-30T13:00:00+01:00

It’s that time of the year. Christmas holidays. So I thought to myself. Why be naughty this year? This year I’ll be a good boy.

Jææææs I will (as if :stuck_out_tongue_winking_eye:).

I decided to optimize security and a bit of privacy for the websites I’m hosting. Namely:

The bengtssondd blog. This site.
Skakmat digte. A website for my poems.
Retshjaelp. The personal website of the lawyer Kaaveh Piroz.

After having optimized various HTTP security headers for the above websites I wanted to enable OCSP stapling for them as well. Now, this was more of a challenge. I needed to leap over several obstacles.

They where:

Understanding OCSP stapling
Configuring HAProxy to use it
Automatizing the continued use of OCSP stapling for the certificates used by the websites in question

OCSP stapling

OCSP is an acronym for Online Certificate Status Protocol. It is a way for website servers to save on network IO and performance. A way to increase the privacy of your website users. And it makes your website less vulnerable to failing or overloaded CA provider infrastructure.

OCSP gives you these advantages by:

Taking away the need for end-users to issue calls to the certificate issuer. By offloading certificate revocation calls to the website server
- Resulting in more privacy to the end-user in that their visit to the website is not revealed to the certificate issuer
- Saving performance for the end-user as their browser will not have to make the lookup
- Websites loading a tiny tad quicker for the end-user than without OCSP stapling
Saving on the number of necessary certificate lookup calls to the certificate issuer. As an OCSP file is saved for each certificate on the server and data from that file is presented to visiting users
- While not actually saving performance for the web-server, it is implemented in a way that is well thought out, in regards to lessening the performance burden
- CA provider infrastructure is a known target for ill intended people. So by caching OCSP responses a website is less vulnerable to DoS attacks on CA’s

Who doesn’t want that?

Configuring HAProxy to use OCSP stapling

HAProxy is the load-balancer fronting the mentioned websites. Let’s configure it.

All HAProxy requires in order to use OCSP stapling is:

An .ocsp file. Named as the crt/pem file it is covering. With the sole addition that it should have the .ocsp extension
- The OCSP file for e.g. my-certificate.pem has to be named my-certificate.pem.ocsp
The .ocsp file should be valid and in the DER format. To be valid:
- it has to indicate a good status
- it has to be a single response for the certificate of the PEM file, and it
- has to be valid at the moment of addition

More here.

Readying HAProxy

The stats socket will be utilized for the automatization of HAProxy. This socket is not enabled by default. You need to do the following:

Enable the stats socket in the haproxy.cfg file

    stats socket /haproxysocket/haproxy.sock mode 660 level admin
    stats timeout 2m

add the above to the global section in the haproxy.cfg file.

Reload/restart HAProxy for the above to take effect

N.B. If you run HAProxy via a Docker container you need to do something along the lines of:

docker run.... -v /folder/to/contain/the/socket:/haproxysocket/
the folder referenced in the -v specification above is the same folder you use in the script below. So by referring to /folder/to/contain/the/socket/haproxy.sock in your socat cmd you will be “talking” to the socket inside the HAProxy container.

Generating the OCSP file

So how does one generate the OCSP file. You dig around and find someone like Igor Cicimov. He made a nice script for generating the OCSP file and configuring HAProxy with it. See his blog post. However I had to update the script a few places.

This part was not working on my QNAP: ISSUER_NAME=$(echo ${ISSUER_URI##*/} I had to change it to ISSUER_NAME=$(echo ${ISSUER_URI##*//}.

And because HAProxy did not like the issuer certificate to be in the same folder as the actual website certificates (because the issuer certificate do not contain a private key), I had to update -out ${DIR}/${ISSUER_NAME}.pem on the wget and openssl ocsp commands to /tmp/${ISSUER_NAME}_CHOOSE_YOUR_POSTFIX.pem.

I also removed [[ $? -eq 0 ]] && [[ $(pidof haproxy) ]] && [[ -s ${DIR}/${CERT}.ocsp ]] && from the set ssl ocsp... line. As I don’t see the use for it and don’t fully grasp what it is trying to do. Feel very free to ping me if you are able to decipher this black magic :smile:. Find my Twitter under “About” in the lower left corner of this site.

After the above changes the script works. However, if you have several certificates and try to run the script for another certificate, the set ssl ocsp..... command will fail, if there isn’t already an OCSP file for the respective certificate. The error will be: OCSP single response: Certificate ID does not match any certificate or issuer. To alleviate this you’ll have to generate the OCSP file for all your certificates and reload HAProxy. The above issue apparently happens because of the way that OCSP support for HAProxy was designed. Its initial purpose was to update the pre-loaded OCSP record. So if there is no OCSP file for a certificate already, and you execute the set ssl ocsp..... command, the mentioned error will be thrown.

The leftovers

With the OCSP file generated and the ability to reload HAProxy when the OCSP file has been updated all there is left now is to do this automatically.

Simply setup a cron job. Timing it to once a day should suffice.

When your are satisfied with your setup and want to see the fruits of your effort. Go-to SSL Decoder and give it a spin. Under OCSP Stapling, in the result table it should say Cert status good and so forth.

Congrats on using OCSP stapling. We can all be happy with that.

Useful resources

Thank you for reading along.

Over and out :dash:

When you are lucky good. A story on a rotten character. That trashed .netrc and therefore ConcourseCI resources.

2018-10-16T00:00:00+02:00

This is a journey into the world of troubleshooting, pulling out ones own hair and coming out wiser and happier.

So I had this issue:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
Cloning into '/tmp/semver-git-repo'...
fatal: could not read Username for 'https://bengtssondd.it': No such device or address
error checking for new versions: exit status 128`

This happened when the semver (semantic versioning) resource of the CI product named ConcourseCI tried to poll my private Git backend. It was rather frustrating, as it meant my tirade into setting up a proper build pipeline for the HealOps project went into a total halt. Furthermore, the same error occurred for the git ConcourseCI resource.

Then what is a man to do? Find the culprit of course. Then smack it in the face….like…kapow!

So I tried

Downgradring to v3.10 of ConourseCI where it used to work. It still didn’t.
Thinking about what I had changed in my ConcourseCI setup.
- I had “upgraded” the docker-compose file to v3.2.
- Updated the Postgres SQL database backend used by ConcourseCI to v10.5.
Then I tried setting up ConcourseCI, the HealOps build pipelin on another Macbook I have. And no error. It works. But why in the world.
- Then I considered what the difference between the two Macbooks is. The MacBook where it works do have a newer version of Git installed. But could that really be it? Does the semver and git resources really use the local version of Git? What is going on here? It would be great to get a clearer view on this issue.
I followed that line of thought and updated Git on the non-working Mac. I even did a full Docker restart on MacOS. And a full re-install of ConcourseCI (I also removed the Postgres docker volume). The issue remained.
Then I saw someone mentioning the fly hijack command that ConcourseCI have. I went ahead and “hijacked” the worker container in which the semver resource is running. I executed git clone https://bengtssondd.it/git/healops.git to test git and was prompted for a username and password. When entering those two correctly the git clone command executed without problems. The issue still remained though.

Hmmm. Now what. What do you do when you are in the center of the mist are kinda lost. You poke around. Try out new routes. Wave your arms. Find your smartphone flashlight.

So I poked around while in fly hijack mode. Did a little ls -la in various places. Aaaand what was that. In the root of the container filesystem there was a .gitconfig file. Let’s cat .gitconfig that. Hmmmm nothing in there of interest. Oh my. There is also a .netrc file in the. Let’s hit that with a cat .netrc as well. Aaaand what do we have in there my friend?

We had.

default login myusername password \!MyPassword((88))

Immediately I noticed the leading \. It shouldn’t be there as it is not a part of the password. It is there in order to act as a escape character to the ! character. Necessary when executing the fly -t myConcoursePipeline set-pipeline --config ./pipeline.yml -p PipelineName --var myusername=\!MyPassword((88)). If not the ! character will be interpreted as a special-character and not as a regular character that should be a part of the value to the myusername variable. As soon as I saw that I removed the \ character via Vi. Then tried my git clone https://bengtssondd.it/git/healops.git command again. This time with no prompts. I think we are there. A nice bubbly feeling started to ooze in my chest :yum:. I looked up .netrc and I was further re-assured in this being the culprit. .netrc is a part of the Linux auto-login system. It is used for providing login info automatically as the name implies. I tested by updating the HealOps pipeline yml file » comitted it to ConcourseCI with the fly -t myConcoursePipeline set-pipeline.... command. Waiting … tic … toc … tic … toc :watch:.

Ta-da freaking laaaaaa. It worked. That was it. A special-character f** up my life. However, nice that I was able to half troubleshoot myself en route to the culprit, half stumble over it.

Now I just have some failing Pester tests when HealOps is building. More on that in a later post.

For now, onto filling out a bug report on the Concourse semver and git resource GitHub pages. I think special characters should be supported. The bug report for reference.

Thank you for reading along.

Over and out :dash:

HealOps, the healing monitoring framework. Part 1 - Why HealOps?

2018-08-11T19:00:00+02:00

Intro and info.

The diagram above is a context diagram over HealOps. It displays the components that are a part of HealOps. What they deliver to the system and whether they input or extract data in relation to their role in the system. Finally the actors of the system can be identified.

HealOps is a monitoring and healing framework, developed primarily in PowerShell, for IT systems and their components. I designed and developed it with the motivation of getting fewer on-duty calls, save time and as a way to reach a system uptime that is as near 100% as possible. The thought is that by creating a system, that uses the TDD (Test Driven Development) process. You have a clear and formalized set of rules to test the state of an IT system and its components. This then makes it possible to act and repair an identified broken system state. The identification and repairing of a broken system state happens automatically. Throughout this cycle of testing and potentially repairing state, state data is automatically reported to a Time Series Database. State data can then be pulled into a monitoring and visualization system like Graphana. From Grafana and other systems like it, it is possible to integrate with an incident management system. With this setup a potential broken state of an IT system, that could not be fixed, is reported to Grafana, which in turn sends a request to an incident management system. Then the incident management system can be configured to automatically trigger an alert to the on-call duty personnel. Thereby getting us some human eyeballs on the issue.

A visualization of the above description of state monitoring and repairing will likely help on understanding HealOps. A sequence diagram comes in handy.

The above image shows the main functions involved when testing and repairing the state of an IT system component.

For everything related to HealOps, look now further:

Code > here
News > Right here where you are. My Bengtssondd blog.
The occasional update > Twitter

Spill the beans. What are the benefits of using HealOps?

You save time by reducing the number of times where you have to “pull the chestnuts out of the fire”.
You save up energy and lust for fixing and improving instead of just maintaining and keeping above water.
Monitoring and healing via HealOps happens automatically. A scheduler is used as a facilitator for running jobs that monitors the state of one or more components of an IT system. Thereby reducing the reaction time from identifying a broken state of a service until a repair of that service kicks in.
What is being done and what was done is clear to everyone.
- How and what is tested is clearly formalized in code in a test file.
- State data is reported to a monitoring and visualization backend system.
- Reports can be generated on state data. As well as on the cases where an alarm was triggered for human intervention.
- HealOps logs what it does along the way.
When you are on-call duty and get an alarm. The info you get along with the alarm is at a high detail level. As you can send along a screenshot of what triggered an alarm in Grafana. Also, it is possible, in most incident management systems to specify information to send with a triggered alarm. This could be a wiki article, tips and tricks, what often resolves a given case and so forth.
HealOps could save you money on personnel that is employed only to glance at screens in order to react in case of a state going bunkers. As HealOps automatically identifies a state, very likely quicker than a human, and then automatically tries to repair the broken state.
- The money saved could then be used on personnel that improves the product you sell. The end-customer experience. Company facilities and so on.
It is open-source and free.
- You can review the code.
- Help and commit new features and fix bugs. And thereby speed-up the delivery of the features you would like to have included or the bugs that needs to be fixed.

Any downsides?

Fair question. Here’s the ones I’ve been able to identify so far.

Only one Time Series Database backend is currently supported. Namely OpenTSDB.
- It’s the plan to support several more over time.
There is only a few HealOps packages currently available. A HealOps package is the entity that holds Tests and Repairs files. A Tests file is invoked in order to verify the state of a component of an IT system. If its state is broken it will be tried repaired via the Repairs file. As you can understand, these packages are both an integral part, as well as very useful. So you will have to do a little bit of work here, in order to be able to us HealOps together with the IT systems you have. Luckily I’ve designed the way to develop a HealOps package so that it could be standardized and easy to follow. So with a little bit of practice and by reading the (upcoming) documentation you should be able to do like a boss.
Running HealOps on Linux and MacOS is right now not officially supported. But it is the number one priority. The plan is to support it within the next 2-4 months.
As HealOps is developed in PowerShell it is not extremely performant. However, I’m considering refactoring parts of HealOps. Meaning that HealOps, in the future, might become a binary PowerShell module.

HealOps needs a back rub.

There are several features I would like to implement in HealOps. Number one on the list is job scheduling on Linux and MacOS. But there is more. Full testing of the HealOps code. In Pester. Documentation. Considering supporting container virtualization. Creating prototypes for that….and the list goes on. But time is flying.

So, if you think HealOps is a cool system and the type of project you could see yourself spend time on. Be my guest. I would highly appreciate it. If you have any questions or need more info, before you commit yourself. Feel free to hit me up on Twitter or LinkedIn (links for those services can be found in the lower left corner on this blog).

Coming up.

In part two on HealOps, I plan to describe the components of HealOps itself in great detail. The technical nerdy stuff. I hope you will read along when that comes out.

Until then you are more than welcome to have a look at the code. Find it here.

Adieu.

I hope you enjoyed this post and that you will checkout HealOps. Thank you for reading along.

Over and out :dash:

Wording my graphing

2018-08-05T00:00:00+02:00

I bet you know the feeling of running Install-Module -Name SomeInterestingModule, waiting, then, with your fingers on the keys, so ready to go play with SomeInterestingModule just downloaded. You stop. Pause. Thinking to yourself: where do I start with this module?

Okay, you could run Get-Command -Module SomeInterestingModule and then Get-Help Get-CommandInSomeInterestingModule. However, the module would still be a blackbox. This is where my function Out-PSModuleCallGraph comes into play. It helps you dive into the inner workings of SomeInterestingModule. The commands the public commands of the module are calling, what those commands are calling and so forth. In other words. A way for you to get a look behind the scenes. And thereby an idea into which commands to go-to in specific situations.

Here is some goodies on how to use Out-PSModuleCallGraph and the experience I had developing it.

Use it!

Run Save-Script -Name Out-PSModuleCallGraph -Path "PATH_WHERE_YOU_WANT_TO_STORE_THE_FILE" (you could also use Install-Script. But with Save-Script you have more control and can look through the code (for security reasons))
1. Pre-requisites:
  - A connection to the Internet.
  - The PSGallery should be registered for the PowerShellGet module to use (Guide to the PowerShellGallery).
  - PowerShell 5.1+ on your box. (PSCore if on Linux or MacOS).
Dot-source the script. E.g. . "PATH_WHERE_YOU_STORED_THE_FILE". When you want to use it.
- Here is an example of dot-sourcing > . ./usr/local/share/powershell/Scripts/Out-PSModuleCallGraph.ps1.
After you have dot-sourced the Out-PSModuleCallGraph script, the Out-PSModuleCallGraph command is available in your PowerShell session. So now execute Out-PSModuleCallGraph. E.g. Out-PSModuleCallGraph -ModuleName Pester -ShowGraph. That cmdline will generate a call graph on the Pester module and show you the Graph after it has been generated.

Learning outcomes of developing Out-PSModuleCallGraph.

The best thing about programming is that you learn so much and it gets you deeper into the labyrinth of how stuff works in the world of IT. Not the least, it strongly requires your talent for being creative. It’s just b e a utiful…..Allllllrighty then!!! Let’s go. In the below I will share learnings and tips that I got out of developing Out-PSModuleCallGraph.

Attacking the problem.

In this case it was all about identifying the info I needed to derive from a PowerShell module being analyzed in order to be able to generate a call-graph on the module. I came up with the following list.

Identify the public functions.
Identify the private functions.
Somehow, parse these functions in order to infer what commands/functions they use.
Find a way to tag the commands each function uses. So that it can be identified what the scope of the command is. Meaning. If it is a public command in the same module. A private command in the same module or an external command, so not a part of the module.
Register the chronological order of the commands called in the functions.

Let’s concentrate on the tough challenges > Identifying the private functions and how to tell the boundaries of a function via code.

How to tell where a function ends

Yeah how exactly do you do that? Especially when you can’t glance at the code with those preying eyeballs. The solution I came up with, was to count brackets (these guys > {}). Funny how something you learned when walking your baby programmer shoes comes in handy later on. So the story goes…..When I started out programming and was so foolish not to use a proper IDE. It was before the days of VS Code, Atom and the like. You could somewhat easily end-up “forgetting” to end a function declaration or so. You would then get a run-time error along the lines of ...missing closing bracket }..... So now you had to figure out where this ending bracket was missing. You then had to count from the starting bracket. Add to your brain memory counter every time you encountered a starting bracket in the code. Detract when a closing bracket was encountered. You then knew you had the closing bracket when the calculation was at a nice round 0. So I basically programmed a function to use with Out-PSModuleCallGraph that calculates the boundaries of a function by counting brackets. Lovely!

Identifying private functions

Identifying the public functions is quite easy. It’s just = Import the module being analyzed » then use the Get-Command Cmdlet to get the public functions of the module.

The private functions though, that is quite different. A private function is … yeah … private. So there is nothing else to do than parse the code of every file in a PowerShell module » determine the declared functions in the files » look into a list of the public functions » if the function is not in the list » it is private. It takes some time and requires juggling around with several collections. But that’s how I did it. You might have asked yourself. What does this dude mean by “…parse the code….” in the above. Let me explain. What is really going on there is e.g.:

# Tokenize the content of the file
$ast = [System.Management.Automation.PSParser]::Tokenize( (Get-Content -Path $PS1File.FullName), [ref]$null)

# Identify the declared functions in the file. Goes from top to bottom. So the first declared function will be at idx 0
[Array]$DeclaredFunctions = $ast.where( { $_.Type -eq "Keyword" -and $_.Content -eq "function" } )

The code above is from the code section in Out-PSModuleCallGraph where private functions is being processed. A PS1 file is read » the internal .NET class *.Automation.PSParser is being used to divide the code in the file into AST tokens. AST stands for abstract syntax tree and is something you can read more about here. It is really powerful so give the link a friendly click. I then, throughout the code use the .Where() method of the AST object to filter on the AST. In order to e.g. get me the declared functions, the AST tokens containing brackets and so forth. What a joy / the AST is my toy.

Tips and tricks

Tips on PowerShell parameter validation

I don’t often see people validate collections. It is the nice thing to do :smile: and it is easy. E.g.:

[Parameter(Mandatory, ParameterSetName="LinesToExclude")]
[ValidateScript({$_.Count -ge 1})]
[System.Collections.ArrayList]$LinesToExclude

validates that the collection of type ArrayList actually contains members. I would say that it is good coding style to fail early. Especially in regards to parameter validation. Whenever you can do that early, please do! It helps the end-user and makes your code cleaner, as some conditionals can be avoided in the code.

Another one is to be specific on the type of the parameter. PowerShell is not a strongly typed language. However, as for many other programming languages it helps to declare the type when declaring variables. The IDE you use can infer much more on the code you are writing. And you will be helping the end-user by declaring the type of a parameter. As PowerShell will be able to tell the end-user that the value they are trying to feed into a parameter is wrong and state what the type needs to be. As you can see by the below example you can get pretty crazy when specifying the type. So be as specific as needs be.

[Parameter(Mandatory)]
[ValidateScript({$_.Count -ge 1})] # Validate that it is not an empty collection.
[System.Collections.ObjectModel.Collection`1[System.Management.Automation.PSToken]]$AST

The type declaration is the [System.Collections.ObjectModel.Collection1[System.Management.Automation.PSToken]]$AST part.

Let me use Out-PSModuleCallGraph to exemplify.

I followed the guide under Use it!
Ran > Out-PSModuleCallGraph -ModuleRoot ./FranzJaeger/FranzJaegerClient/ -OutputFormat jpg -ShowGraph. Which gave me:

Wonderful. What can we reason from that picture.

We can quickly get an overview over the number of public functions in the module.
The names of every function in the module.
What calls what.
Which external commands are used.
The chronological order by which a function calls the commands it uses.

Kudos

Thank you Brian Bunke. Your publishing-scripts blog post helped me publish the script to the PSGallery.
Thank you Kevin Marquette. For developing PSGraph. Out-PSModuleCallgraph uses PSGraph to generate the graph after analyzing a PowerShell module.
Thank you Mikkel Rømer. For discussing this idea with me and for giving me some tips on how to possibly attack the challenge.

Postludium

Thank you for reading along. I hope you learned something and that you can use the Out-PSModuleCallGraph script/function.

Nota bene. If you bump into a bug or have a feature request feel free to report them to the issues section on this GitHub repository PowerShellTooling-Github.

Over and out :dash:

Publishing the blog with ConcourseCI

2018-04-21T00:00:00+02:00

Okay. The blog is up and running (yeah it’s alive). With that of the bucket list on to the next work item on the list: Build and publish. My process of publishing new posts and generating the website with Jekyll, has up until now been quite cumbersome. Furthermore, some time ago I heard about a piece of CI software called ConcourseCI. I felt so enthused about it that I wanted to try it out. See if it could make it work for my own software pipelines as well maybe use it at work.

How I made publishing of my blog easy with ConcourseCI will be the spicy ingredients of this post.

ConcourseCI - yeah what about it?

ConcourseCI is the work child of Alex Suraci and Christopher Brown. They, and a team of engineers has, for some years, been sponsored by Pivotal. A spin-out from EMC and VMWare. And a major player in the CloudFoundry project. ConcourseCI, as the name implies, is a continuous integration software product. Its advantages and what separates it from its competitors are:

Build config is entirely codified. Nothing is in a hard to find sub-menu in a GUI.
Can run in Docker containers.
Builds are always atomic.
- Builds run inside a container (the virtualization technology), spun up to execute that single build.
Build ‘x’ cannot effect build ‘y’. As builds run inside a sealed of container you won’t have e.g. a global variable spill into the next build. That could be a serious issue if that build also uses the variable.
It is very comforting that you can stand to loose your build server and its setup. As all you have to do is spin-up your ConcourseCI setup once again, which you can do easily. Because you:
- Used technology such as Docker Compose or Docker Stack.
- Used Infrastructure As Code (I like to call it “Infrastructure From Code”).
- Described your build pipelines in YAML. So you’ll just have to push them to ConcourseCI, when it’s up again after a breakdown and you are good to go.
It has a very friendly community at least to the extend I have been involved with it. And that is around five-six times the last month.

Some of its disadvantages are:

Its a rather new CI product on the market. So it hasn’t been battle-tested in a great number of companies. However, it is part of some major software build pipelines. E.g. at Pivotal.
At least for me, was an entirely new way of designing your build pipelines. Learning the syntax of ConcourseCI to write the required YAML for a Jekyll build required some trial and error and a lot of reading. It takes time to get around the concepts of ConcourseCI. But that’s how it goes whenever a new kid arrives on the block. You have to get acquainted with the kid first before you find out that, yes they are actually befriend-able.

ConcourseCI, let’s get at it.

What I wanted to accomplish with ConcourseCI

Primarily to get a way to automatize and formalize the build and publish of my Jekyll website (the one you are reading on right now). However, I especially wanted to find out if ConcourseCI could hone up to the impressions reading about it gave me.

In work-items this measures into:

Trigger a build whenever a push to the git repository, holding the bengtssondd.it code, happens.
Use semantic versioning to bump builds and thereby software releases.
Generate bengtssondd.it with the jekyll build --destination /PATH/ cmd.
Notify myself on telegram, when the build is done, with my own little telegram bot.

Installing it

Initially I wanted to run ConcourseCI on my QNAP NAS. However, I wasn’t able to make it work. I created a case with QNAP support. Waited, pinged them for status updates again and again. I have to say. Their response time is certainly not impressive. However, they did end up giving me the cause of the issue. The conntrack kernel module is not enabled in the current QNAP firmware version. It will be for the next one == v4.3.5. Yay! That won’t help me now, but it’s always good to know the whys. So. I ended up running ConcourseCI on my Mac.

The nitty gritty details of the conntrack issue (feel free to jump this)

It was the ConcourseCI worker container that did not want to start up. It would just throw the error concourse-worker_1 | iptables: No chain/target/match by that name. I sought help on the ConcourseCI Slack channel and got plenty. That helped me identify the source of the error. Which was this line in the CloudFoundry Guardian software. So this script tries to create some iptables rules in order for the ConcourseCI software to work as intended network wise. But! Because the conntrack network kernel module isn’t enabled on my QNAP right now, this fails.

I used the following docker-compose.yml file to run ConcourseCI on my Mac.

version: '3'

services:
  concourse-db:
    image: postgres:9.6
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: 
      PGDATA: /database

  concourse-web:
    image: concourse/concourse:3.10.0
    links: [concourse-db]
    command: web
    depends_on: [concourse-db]
    ports: ["9090:8080"]
    volumes: ["./keys/web:/concourse-keys"]
    restart: unless-stopped # required so that it retries until concourse-db comes up
    environment:
      CONCOURSE_BASIC_AUTH_USERNAME: concourse
      CONCOURSE_BASIC_AUTH_PASSWORD: 
      CONCOURSE_EXTERNAL_URL: http://IP_OR_DOMAIN:9090
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: 
      CONCOURSE_POSTGRES_DATABASE: concourse

  concourse-worker:
    image: concourse/concourse:3.10.0
    privileged: true
    links: [concourse-web]
    depends_on: [concourse-web]
    command: worker
    volumes: ["./keys/worker:/concourse-keys"]
    environment:
      CONCOURSE_TSA_HOST: concourse-web:2222

And generally followed the ConcourseCI installation guide. Especially the key generating part is important to get right. As a difference from the docker-compose.yml files I found in places like the ConcourseCI website, their Slack channel and the like, I had to explicitly define the SSH port of the TSA Host, that the ConcourseCI worker should use. Notice the CONCOURSE_TSA_HOST: concourse-web:2222 line in the docker-compose.yml file above. This is because I don’t run SSH on the standard port.

The “D’oh!” (/doʊʔ/ DOH) segment to “Installing it”

Please don’t do the same mistake as me. I defined a password for my generated SSH key. As ConcourseCI uses the generated SSH keys to setup an SSH tunnel and does not use password prompting it won’t work if you define a password. I had to bow my head and go back to the cmdline to re-generate the SSH keys. But, hey I got wiser by troubleshooting why the *€#&%/&/””!€%# thing wasn’t working.

Configuring it

The ConcourseCI pipeline YAML file

I made myself a fine little YAML file for my ConcourseCI pipeline. You can call it whatever, but a saying name is likely going to help you in the long run. So please contain yourself. I called mine: bengtssondd-pipeline.yml. I won’t go through the entire file. Rather I’ll describe the concepts on which I stumbled and the issues I had when configuring ConcourseCI.

input / output and paths

ConcourseCI has the concept of input and output.

Names in your output definition will make ConcourseCI create directories in which your build task should store artifacts for later steps to digest.

Inputs uses outputs from others steps or resources. The names you use for input and output will logically be the root folders or at least part of your folder path to be used in other elements of your build pipeline. E.g. if you have a resource named repo (that gets you the commit in a repository), that you use in a get step and therefore use as input for your task. Then you should also use this name as part of the logical path in e.g. a build script. Keep this in mind at all times, when working with ConcourseCI. Because it is such an integral part of building your pipeline and using artifacts between job steps.

Now seeing is believing, at least for most people, so let me finish this section by providing you with the slightly editedbengtssondd-pipeline.ymlfile. Of course I removed passwords and other telling data bits.

resource_types:
- name: rsync-resource
  type: docker-image
  source:
      repository: mrsixw/concourse-rsync-resource
      tag: latest
- name: telegram-notification
  type: docker-image
  source:
    repository: w32blaster/concourse-telegram-notifier
    tag: latest

resources:
- name: status-message
  type: telegram-notification
  source:
    bot_token: ((tgram_token))
- name: repo
  type: git
  source: 
    branch: master
    password: ((qgit_pass))
    uri: https://URI_TO_GIT_REPO/myRepo.git
    username: qgit
- name: version
  type: semver
  source:
    branch: version
    driver: git
    file: version
    initial_version: 1.0.0
    password: ((qgit_pass))
    uri: https://URI_TO_GIT_REPO/myRepo.git
    username: qgit
- name: sync-resource
  type: rsync-resource
  source:
    base_dir: /PATH_TO_WEBROOT/FOLDER
    disable_version_path: true
    port: 2222
    server: IP_OF_THE_SERVER
    user : someUser
    private_key: |
            -----BEGIN RSA PRIVATE KEY-----
            NOT_GOOD_BAD_TO_HAVE_KEY_HERE_BUT_THAT_IS_HOW_IT_IS_RIGHT_NOW
            -----END RSA PRIVATE KEY-----

jobs:
- name: publish-bengtssondd
  plan:
  - get: version
    params: {bump: minor}
  - get: repo
    trigger: true
  - task: generateSite
    config:
      platform: linux
      image_resource:
        source:
          repository: jekyll/jekyll
          tag: builder
        type: docker-image
      inputs:
        - name: repo
      outputs:
        - name: build-generated
      run:
        path: repo/ci/build/generate_site.sh
  - put: version
    params: {file: version/version}
  - put: sync-resource
    params: {"sync_dir" : "build-generated" }
  - put: status-message
    params:
      chat_id: "id_of_telegram_user"
      text: "MESSAGE TO SEND"

The “D’oh!” (/doʊʔ/ DOH) segment to “Configuring it”

Notice theprivate_keyproperty in the pipeline YAML file. That is not good. You should not have keys, password and the like in files that commit to a repository. That is accepted knowledge. Why did I do it then? Because this is a hobby project. I’m the only one with access to the key and the repository. So it’s okay for now. But only for now. On the positive side. Then I have something to look into for an upcoming blog-post. Because, believe me! I have noted that this is something I have to look at.

Pushing the pipeline to ConcourseCI

Fly. Yeah let’s fly. As in hit the skies with the ConcourseCI cli AKA fly. Fly is a very integral part of ConcourseCI. As pointed out earlier in this blog post in the ConcourseCI - yeah what about it? section. When you feel confident and ready to throw your pipeline YAML file at the ConcourseCI backend do it like they do it where they do it, like this:

Install fly.
1. Go to your ConcourseCI backend root landing page. You’ll find the below.
2. Download the fly client for your OS.
3. Put it in whatever folder.
4. Make sure to put the folder in your OS environment PATH variable. You’ll know you have done it correctly when you can open up a new cmd/bash window and execute fly --version and get back a non-erring result (a version number).
Register the ConcourseCI backend.
1. fly --target "NAME_TO_GIVE_THE_TARGET" login --concourse-url http://"DOMAIN_OR_IP":"PORT"
Login to the ConcourseCI backend.
1. fly -t "THE_NAME_YOU_GAVE_THE_TARGET" login. You will be prompted for a password and a username. This is the one you specified when installing the ConcourseCI backend.
Push the pipeline.
1. fly -t "THE_NAME_YOU_GAVE_THE_TARGET" set-pipeline --config /PATH_TO_YOUR/pipeline.yml --pipeline "THE_NAME_TO_GIVE_YOUR_PIPELINE" --var qgit_pass="PASSWORD_TO_GIT
Notice the--var qgit_pass=****part of the fly set-pipeline cmdline. This is a great feature of ConcourseCI, that helps you avoid having passwords in your YAML file. So please use it :wink:.

Pro-tip! > use the\character to escape special characters in e.g. a password.

If you have followed along with vigour and a bright smile on your face. You will be able see something similar to the below on your ConcourseCI backend.

Specifics about the resources used in the ConcourseCI pipeline.

semver

Do remember to create a branch. Allow me to suggest you to name it “version” or something similar. As this will store the release version of the ‘thing’ you build.
- If you don’t you will get a > fatal: Remote branch version not found in upstream origin type error.
If you have a > SSL certificate problem: unable to get local issuer certificate error checking for new versions: exit status 128 type problem you should likely look into:
- that the entire certificate chain can be downloaded. Use the openssl cmdline tool to troubleshoot that. A laopenssl s_client -connect "troubling_domain":443 -servername "troubling_domain" -showcerts
- that you don’t have a LetsEncrypt certificate that was generated in a way that malformed it. Sorry I can’t be more specific as of now.

git

Use a proper certificate and ensure that the entire chain can be verified and downloaded. Use the same tips mentioned in the semver section above.

telegram

Getting the chat id. The easiest way I found was to:
- Create the bot by adding @BotFather to your Telegram CLI.
- Follow the instructions in order to create your bot.
- Send a message to your bot. Just some random crap. It doesn’t matter. this step is important! if you don’t execute it the response from the API endpoint below will just be an empty array/[].
- Execute https://api.telegram.org/bot'API-TOKEN'/getUpdates where the ‘API-TOKEN’ part is the token of your bot you just created via BotFather.
- In the JSON returned you will see a chat id in the chat section. That is the one you want to specify in the chat_id property in your pipeline YAML.

Learning outcomes. School’s out.

Eat your breakfast. Everyday. Yes! Everyday. Ooooops sorry. This is not some freakin’ paleo blog. Then again. We can all use a foodie tip now and then.
Seriously. Read the documentation on the software products you are trying to learn. Here is the good sources that helped me during my travels through the world of ConcourseCI.
1. https://github.com/concourse/docker-image-resource
2. https://concourse-ci.org/docs.html
3. https://github.com/concourse/git-resource
4. https://github.com/mrsixw/concourse-rsync-resource
5. https://github.com/w32blaster/telegram-notification-resource
6. https://wiki.qnap.com/wiki/SSH:_How_To_Set_Up_Authorized_Keys for connecting over SSH with Rsync.
7. https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted#668049 helped me in finding out why the files could not be written to the destination folder. It was a permissions issue. Owner had to be the user running the rsync job.

Thank you for reading along. Also a thank you to the kind people that helped me along my journey of getting this to work. They are:

Alex Suraci. One of the initial creators of ConcourseCI.
Christopher Brown. One of the initial creators of ConcourseCI.
Abraham Cabrera. A friendly ConcourseCI user on the ConcourseCI Discord need-help channel.
Morten Frydensberg. He’s a DevOps Engineer at Danske Spil and was so kind to listen to, both my clever and stupid ideas, as well as to troubleshoot the ConcourseCI worker container issue with me.

Over and out :dash:

Booting up the blog

2018-03-19T00:29:48+01:00

Hi there and thank you for visiting Bengtsson Driven Development. My name is Lars Bengtsson and I’m currently working for Danske Spil as a DevOps Engineer. For quite some time I have thought about starting up a blog. However, throughout that process all the hows, can I’s and so forth, started to ooze out from my brain. But then, a few months back, as I wandered about on the interwebs I bumped into this static site generator called Jekyll. Not only hope came to me. The feeling of triumphant victory as well. I quickly hit up Ctrl-D to have myself a bookmark…..aaaaand went back to my regular routines.

Many a day went by. Finally I conquered my usual process of ‘let me do this other “thing”’, ‘let me do that “thing” right here’ and went about it. So without further ado…..

Wassup Jekyll

“Say hello to my little friend” AKA Jekyll. A ruby based static site generator. You write your posts in markdown. Oh wee. Easy peasy, nah think again (at least I would come to think better of it). However, it reads to be very well suited for the CI/build happy people out there ;-). You know, markdown/static text. Can I program a parser for that? Yes you can! A converter then? Yes you can! Hooray, let’s:

go make the web great again.

Or is it US of A? Anyways. If nothing else, GitHub is pretty happy about Jekyll. So much that you can use GitHub as a webserver for your Jekyll based site (see: GitHub Pages). Furthermore, there seems to be a good community around Jekyll, its free and several pre-cooked, customizable themes are available. These are some of the features that made me realize that Jekyll was exactly what I had been looking for. So please, come a long for a trip down memory lane.

…as I go back into my short-term memory. Through the process of launching this very website/blog.

Setting up Jekyll

In bulletproof bullet points the frog croaked.

A good old follow this how-to was my friend.
Finding a theme was needed, so I gave this web-query a hit-up (hit-up explained, if needs be.).
Now lets read that Hydejack theme documentation.
1. And some more Jekyll doc.
To the SoThem links, okay SoMe, but isn’t it really about the social media companies themselves, at the end of the day? You tell me!? Oh hey, so what I did was to make sure that URI’s to twitter, GitHub and so forth, pointed to the correct SoThem websites.

Customization

- pictures

I went looking in my digital archives and found a good picture for the left sidebar. The picture is from Japan. It’s a Jellyfish my friend, and the pic was taken at Aqua Park Shinagawa. Then one of myself. With glasses, even though I don’t use focal aids. But hey, I call myself a connoisseur, so the least I could do is to pretend to look like one.

- the favicon

A had a little gem laying around. So I used that. If you can guess/see what it is I will give the very fine price of a digital high-five. What, to you, is probably more interesting, is how I overruled the favicon that the HydeJack theme uses. So….

A general rule is that a Jekyll theme has an “assets” folder. In this folder the theme author would store images, favicons and so forth. I therefore created such a folder in my websites local Git repo.
Inside the assets folder I created a folder named “icons”.
You guessed it? Store the favicon you want to use inside the “icons” folder. And! Make sure to name it “favicon.ico”.

Pro tip! In order to see the folder structure of the theme you are using. Execute > open $(bundle show minima) on macOS© or > explorer /usr/local/lib/ruby/gems/"Ruby_Gems_Version"/gems/"theme_name_and_version" on Windows©

The first post

Hey there, look at my monkey AKA my first blog post.

In fact you are. Right here.

You write your posts in Markdown when you use Jekyll. What’s more though is the fact that writing in markdown is not intuitively straight forward. Much the same as learning pretty much any other new language, whether that is Swedish (jävla bra) or Brainfuck. Markdown it is though. Markdown has its own syntax and various people have made it fit specific scenarios. However, Jekyll seems to respect the Markdown language at its core standard. What’s the fuzz then? The exact syntax! That’s whats up. How-to do a comment, a link reference and for example a clickable picture without the anchor icon. Which by the way on this HydeJack Jekyll theme, is not mentioned in the documentation.

So I had to:

Open the HydeJack theme folder. As mentioned in the “Pro tip!” above.
Look around, looking for clues on how-to exclude the anchor icon.
1. The “_mark-external.scss file under */jekyll-theme-hydejack-“versionNumber”/_sass/hydejack/__link” looked promising.
I opened it up and found a CSS class named “{.no-mark-external}”.
Added it to the image link:
1. before == [![](/somewhere/image/yesSir.jpg)][linkSectionName]
2. after == [![](/somewhere/image/yesSir.jpg)][linkSectionName]{:.no-mark-external}

The anchor was gone and daddy was happy :clap:{:style=”display: inline”}. Notice that clap emoticon being displayed “inline”? Oh yeah, thank you Jekyll for using kramdown as your converter. As this converter supports setting style related settings in markdown directly. As I did with that clap emoticon.

I had to:

Write > : clap :{:style="display: inline"} in markdown. The : clap : part (normally without spaces) is what is rendered to be the clap emoticon image. The : clap : placeholder works because I have installed the emoji_for_jekyll plugin. It took me a while to figure this out, but I arrived in due time. Documentation, tests and fiery helps A LOT.

Getting the blog on its wings

Hmm now to the question of where to host it. A domain for it and all these other peripheral practical tasks of getting myself a blog.

The domain and a story on the name

Yeah why that freaking name? I mean, come on, Bengtsson Driven Development? Wassup with that! Listen up :smile:{:style=”display: inline”}. Have you heard of Behavior Driven Development (BDD)? Experienced the jolly-go happy developers at Danske Spil? And now what’s my name? What do I do, DevOps. A development driven methodology towards running software platforms and services.

It’s aaaaaaalive and its perfect.

That mesh of influences made me land on Bengtsson Driven Development as the name for my blog. A special thanks goes to Micki and Kasper, two developers at Danske Spil, for opening my drowsy DevOps eyes towards the potential of Bengtsson driven/bentze driven/#kanViFåOsNogetBDD.

That being the name. A little long though, so decided on bengtssondd.it as the domain. .IT < yeah it is the Italian tld but it is also my subject area :computer:.

DNS

From my days in the hosting biz I had come to know of GratisDNS. But, do they support DDNS? Oh yes they do. Do they have an API, I can call to update DNS records for the bengtssondd.it domain? Yes they have. Wicked times :dancers:.

In order to be able to use the GratisDNS DDNS API service you have to setup a DDNS password on your account. See more here
Generate the URI to use for updating the IP for the domain > GratisDDNS.
From a ‘box’ on the same network from where you will host the website (the WAN source address has to resolve to the same address as on the ‘box’ that actually hosts the website.)
1. Query the URI you generated in step 1 with cURL or something similar.
Test that it works.

Seriously consider creating a cron job or whatever so that the WAN IP is automatically updated on the GratisDNS servers.

HTTPS on bengtssondd.it

This 1 was “more” fun. But hey, I learned myself some more HAProxy and Letsencrypt. What I wanted was to have a setup where:

I get a certificate that is approved by a trustable certificate authority and it should be free.
It is possible to renew the certificate automatically.
Ceritificate revocation should be possible.
It should be possible to use with HAProxy.
Also, it should be possible to use more than one certificatae with HAProxy. As I have different websites going through the same HAProxy.

I already had experience with Letsencrypt and it supports my other requirements as well. So I continued with that. Buuuuut! Because I’m using a QNAP NAS to host my websites there is some quirks to handle.

We are talking a Busybox flavored Linux distribution.
The QNAP per default has an old Python install (Letsencrypt likes Python in several CLI versions).
How exactly would I have HAProxy read more than one *.pem file when going over HTTPS?

I really had to put my max GoGoDuck lmgtfy skills to the test. After going back and forth. Testing. Thinking out a way to do ‘x’ and going back to T H E searchengine I arrived at the following recipe for setting up what I wanted.

Recipe for = > 1 website with a Letsencrypt certificate. On a QNAP NAS, through the HAProxy load-balancer

Check your Python version. If below v2.7.6 update or install a newer version.
1. IF you choose to install a newer version, do it throught the QNAP App-center.
2. When installed make sure to use the Python installation in the .qpkg folder on your QNAP. E.g. by export PATH=/share/DATA_DRIVE/.qpkg/Python/src/bin/:$PATH in your scripts or on the cmdline.
Install the Python package manager pip.
1. Download get-pip.py script.
2. Run it.
Install the cryptography Python module: pip install cryptography
Install the python version of certbot, the Letsencrypt certificate cli program.
Check that certbot works by executing: certbot -h. If nothing happens or the cmd fails, make sure that the correct version of Python is in your environment path.
I choose to use the webroot plugin method of Letsencrypt. As I did not want to shutdown HAProxy or any other web-related server. I therefore executed the following in order to get myself a certificate for bengtssondd.it: sudo certbot certonly --webroot -w /PATH_TO_WEBROOT_FOLDER/ -d bengtssondd.it --non-interactive --agree-tos --email myEmail@bentze.klartder
- Of course, I prooved ownership by having ensured that the bengtssondd.it domain points to the WAN that the Letsencrypt certbot request would come from. See dns above.
Then we need to concatenate the certificate files that the certbot cmd above spits out. This is done by executing: sudo cat /etc/letsencrypt/live/bengtssondd.it/fullchain.pem /etc/letsencrypt/live/bengtssondd.it/privkey.pem | sudo tee /FOLDER_TO_STORE_PEM_FILE/PEM_FILENAME.pem
- The order of concatenation is in fact very important.
Now to the haproxy.cfg file.
1. First I edited the line that said bind *:443 ssl crt /usr/local/etc/haproxy/ssl.pem, so that it would now say bind *:443 ssl crt /PATH_TO_PEM_FOLDER/"FILENAME".pem crt /ANOTHER_PATH_TO_PEM_FOLDER/"FILENAME".pem
2. Then I added Bengtssondd.it website acl bengtssondd hdr_beg(host) -i bengtssondd.it use_backend my_back if bengtssondd …to my two frontend sections in the haproxy.cfg file.
3. And ended that of by reloading the HAProxy container.
Tested the site and things where :thumb: up.

The following reads helped me.

Python installation steps > https://community.letsencrypt.org/t/dont-know-how-to-bootstrap-certbot-on-your-os/29905/2
Python pip installation steps > https://pip.pypa.io/en/stable/installing/
Configuring HAProxy > https://serversforhackers.com/c/letsencrypt-with-haproxy

Time for celebration

Yehaaarrr….we are live. #FartPå AKA #PedalToTheMetal. That’s all for now folks. Thank you for reading a long. Come back for the next 1. Where I’ll have a look at some CI and CD for Bengtsson Driven Developement (we might be talking Docker as well…oooh wee). Because right now the CI and CD process is pretty ugly:

Build with bundle exec jekyll serve
scp the files to my QNAP NAS from my dev. machine.
F5 for a refreshing “deploy and pray”.

I think you get the idea. We are talking nasty pastry :suspect:.

Thank you for reading along. Over and out :dash:

Bengtsson Driven Development

When a health check makes its host unhealthy

Initial brute-force debugging

Spearhead debugging

Determining resource limit values

Determining the cause of the resource exhaustion

Solving the cause of the resource exhaustion

Conclusion

The fix

The real fix

HushDNS, can I please get me some DNS privacy

Motivation & background

Action points

Determining resolutions to the action points

Anonymity and logging

Encrypt all the things

dnscrypt-server

dnscrypt-proxy

Integrity and anti-forging

Limit tracking

Executing on the findings

A jaw dropping project name

Resolving the anonymity and logging issue

Blocking DNS requests at the Application layer

Configuring Pi-hole

Kick of the Pi-hole container

Settings for the Pi-hole container

QNAP specifics

Useful docs

Tumbling DNS queries

Considering Anonymized-DNS when you run your own dnscrypt-server

Configuring dnscrypt-proxy

Kick of the dnscrypt-proxy container

Settings for the dnscrypt-proxy container

I said: “don’t log me”

Disabling logging in Pi-hole

Disabling logging in dnscrypt-proxy

Disabling logging in dnscrypt-server

Encrypting all the things

Configuring dnscrypt-server

Using it together with dnscrypt-proxy

Verifying that the dnscrypt-server works

With a DNS query

Verifying the use of Anonymized-DNS

Don’t forge my integrity

How DNSSEC works and some background

Configuring DNSSEC

Verifying that DNSSEC works

DNSSEC documentation

Limiting tracking

DNS Query Name Minimisation

SNI & ESNI, Firefox

PLF AKA Page Load Fingerprints

Signaling without exposing your private phone number

The situation

End goal

Findings

Using Signal Desktop with both phone numbers on the same computer

Linking without a secondary phone

As cheap as possible & registration

Holding onto the phone number

Guidance on achieving the above

Readings I found useful

OCSP stapling with HAProxy. From pitfall to euphoria

OCSP stapling

Configuring HAProxy to use OCSP stapling

Readying HAProxy

Generating the OCSP file

The leftovers

Useful resources

When you are lucky good. A story on a rotten character. That trashed .netrc and therefore ConcourseCI resources.

HealOps, the healing monitoring framework. Part 1 - Why HealOps?

Intro and info.

Spill the beans. What are the benefits of using HealOps?

Any downsides?

HealOps needs a back rub.

Coming up.

Adieu.

Wording my graphing

Use it!

Considering Anonymized-DNS when you run your own `dnscrypt-server`

Using it together with `dnscrypt-proxy`

Using `Signal Desktop` with both phone numbers on the same computer