Publishing the blog with ConcourseCI

21 Apr 2018 in Blog-devops on Concourse, Concourseci, Fly, Ci, Building, Continous-integration, Qnap, Jekyll, Docker, Containerization

Okay. The blog is up and running (yeah it’s alive). With that of the bucket list on to the next work item on the list: Build and publish. My process of publishing new posts and generating the website with Jekyll, has up until now been quite cumbersome. Furthermore, some time ago I heard about a piece of CI software called ConcourseCI. I felt so enthused about it that I wanted to try it out. See if it could make it work for my own software pipelines as well maybe use it at work.

How I made publishing of my blog easy with ConcourseCI will be the spicy ingredients of this post.

ConcourseCI - yeah what about it?

ConcourseCI is the work child of Alex Suraci and Christopher Brown. They, and a team of engineers has, for some years, been sponsored by Pivotal. A spin-out from EMC and VMWare. And a major player in the CloudFoundry project. ConcourseCI, as the name implies, is a continuous integration software product. Its advantages and what separates it from its competitors are:

Build config is entirely codified. Nothing is in a hard to find sub-menu in a GUI.
Can run in Docker containers.
Builds are always atomic.
- Builds run inside a container (the virtualization technology), spun up to execute that single build.
Build ‘x’ cannot effect build ‘y’. As builds run inside a sealed of container you won’t have e.g. a global variable spill into the next build. That could be a serious issue if that build also uses the variable.
It is very comforting that you can stand to loose your build server and its setup. As all you have to do is spin-up your ConcourseCI setup once again, which you can do easily. Because you:
- Used technology such as Docker Compose or Docker Stack.
- Used Infrastructure As Code (I like to call it “Infrastructure From Code”).
- Described your build pipelines in YAML. So you’ll just have to push them to ConcourseCI, when it’s up again after a breakdown and you are good to go.
It has a very friendly community at least to the extend I have been involved with it. And that is around five-six times the last month.

Some of its disadvantages are:

Its a rather new CI product on the market. So it hasn’t been battle-tested in a great number of companies. However, it is part of some major software build pipelines. E.g. at Pivotal.
At least for me, was an entirely new way of designing your build pipelines. Learning the syntax of ConcourseCI to write the required YAML for a Jekyll build required some trial and error and a lot of reading. It takes time to get around the concepts of ConcourseCI. But that’s how it goes whenever a new kid arrives on the block. You have to get acquainted with the kid first before you find out that, yes they are actually befriend-able.

ConcourseCI, let’s get at it.

What I wanted to accomplish with ConcourseCI

Primarily to get a way to automatize and formalize the build and publish of my Jekyll website (the one you are reading on right now). However, I especially wanted to find out if ConcourseCI could hone up to the impressions reading about it gave me.

In work-items this measures into:

Trigger a build whenever a push to the git repository, holding the bengtssondd.it code, happens.
Use semantic versioning to bump builds and thereby software releases.
Generate bengtssondd.it with the jekyll build --destination /PATH/ cmd.
Notify myself on telegram, when the build is done, with my own little telegram bot.

Installing it

Initially I wanted to run ConcourseCI on my QNAP NAS. However, I wasn’t able to make it work. I created a case with QNAP support. Waited, pinged them for status updates again and again. I have to say. Their response time is certainly not impressive. However, they did end up giving me the cause of the issue. The conntrack kernel module is not enabled in the current QNAP firmware version. It will be for the next one == v4.3.5. Yay! That won’t help me now, but it’s always good to know the whys. So. I ended up running ConcourseCI on my Mac.

The nitty gritty details of the conntrack issue (feel free to jump this)

It was the ConcourseCI worker container that did not want to start up. It would just throw the error concourse-worker_1 | iptables: No chain/target/match by that name. I sought help on the ConcourseCI Slack channel and got plenty. That helped me identify the source of the error. Which was this line in the CloudFoundry Guardian software. So this script tries to create some iptables rules in order for the ConcourseCI software to work as intended network wise. But! Because the conntrack network kernel module isn’t enabled on my QNAP right now, this fails.

I used the following docker-compose.yml file to run ConcourseCI on my Mac.

version: '3'

services:
  concourse-db:
    image: postgres:9.6
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: 
      PGDATA: /database

  concourse-web:
    image: concourse/concourse:3.10.0
    links: [concourse-db]
    command: web
    depends_on: [concourse-db]
    ports: ["9090:8080"]
    volumes: ["./keys/web:/concourse-keys"]
    restart: unless-stopped # required so that it retries until concourse-db comes up
    environment:
      CONCOURSE_BASIC_AUTH_USERNAME: concourse
      CONCOURSE_BASIC_AUTH_PASSWORD: 
      CONCOURSE_EXTERNAL_URL: http://IP_OR_DOMAIN:9090
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: 
      CONCOURSE_POSTGRES_DATABASE: concourse

  concourse-worker:
    image: concourse/concourse:3.10.0
    privileged: true
    links: [concourse-web]
    depends_on: [concourse-web]
    command: worker
    volumes: ["./keys/worker:/concourse-keys"]
    environment:
      CONCOURSE_TSA_HOST: concourse-web:2222

And generally followed the ConcourseCI installation guide. Especially the key generating part is important to get right. As a difference from the docker-compose.yml files I found in places like the ConcourseCI website, their Slack channel and the like, I had to explicitly define the SSH port of the TSA Host, that the ConcourseCI worker should use. Notice the CONCOURSE_TSA_HOST: concourse-web:2222 line in the docker-compose.yml file above. This is because I don’t run SSH on the standard port.

The “D’oh!” (/doʊʔ/ DOH) segment to “Installing it”

Please don’t do the same mistake as me. I defined a password for my generated SSH key. As ConcourseCI uses the generated SSH keys to setup an SSH tunnel and does not use password prompting it won’t work if you define a password. I had to bow my head and go back to the cmdline to re-generate the SSH keys. But, hey I got wiser by troubleshooting why the *€#&%/&/””!€%# thing wasn’t working.

Configuring it

The ConcourseCI pipeline YAML file

I made myself a fine little YAML file for my ConcourseCI pipeline. You can call it whatever, but a saying name is likely going to help you in the long run. So please contain yourself. I called mine: bengtssondd-pipeline.yml. I won’t go through the entire file. Rather I’ll describe the concepts on which I stumbled and the issues I had when configuring ConcourseCI.

input / output and paths

ConcourseCI has the concept of input and output.

Names in your output definition will make ConcourseCI create directories in which your build task should store artifacts for later steps to digest.

Inputs uses outputs from others steps or resources. The names you use for input and output will logically be the root folders or at least part of your folder path to be used in other elements of your build pipeline. E.g. if you have a resource named repo (that gets you the commit in a repository), that you use in a get step and therefore use as input for your task. Then you should also use this name as part of the logical path in e.g. a build script. Keep this in mind at all times, when working with ConcourseCI. Because it is such an integral part of building your pipeline and using artifacts between job steps.

Now seeing is believing, at least for most people, so let me finish this section by providing you with the slightly editedbengtssondd-pipeline.ymlfile. Of course I removed passwords and other telling data bits.

resource_types:
- name: rsync-resource
  type: docker-image
  source:
      repository: mrsixw/concourse-rsync-resource
      tag: latest
- name: telegram-notification
  type: docker-image
  source:
    repository: w32blaster/concourse-telegram-notifier
    tag: latest

resources:
- name: status-message
  type: telegram-notification
  source:
    bot_token: ((tgram_token))
- name: repo
  type: git
  source: 
    branch: master
    password: ((qgit_pass))
    uri: https://URI_TO_GIT_REPO/myRepo.git
    username: qgit
- name: version
  type: semver
  source:
    branch: version
    driver: git
    file: version
    initial_version: 1.0.0
    password: ((qgit_pass))
    uri: https://URI_TO_GIT_REPO/myRepo.git
    username: qgit
- name: sync-resource
  type: rsync-resource
  source:
    base_dir: /PATH_TO_WEBROOT/FOLDER
    disable_version_path: true
    port: 2222
    server: IP_OF_THE_SERVER
    user : someUser
    private_key: |
            -----BEGIN RSA PRIVATE KEY-----
            NOT_GOOD_BAD_TO_HAVE_KEY_HERE_BUT_THAT_IS_HOW_IT_IS_RIGHT_NOW
            -----END RSA PRIVATE KEY-----

jobs:
- name: publish-bengtssondd
  plan:
  - get: version
    params: {bump: minor}
  - get: repo
    trigger: true
  - task: generateSite
    config:
      platform: linux
      image_resource:
        source:
          repository: jekyll/jekyll
          tag: builder
        type: docker-image
      inputs:
        - name: repo
      outputs:
        - name: build-generated
      run:
        path: repo/ci/build/generate_site.sh
  - put: version
    params: {file: version/version}
  - put: sync-resource
    params: {"sync_dir" : "build-generated" }
  - put: status-message
    params:
      chat_id: "id_of_telegram_user"
      text: "MESSAGE TO SEND"

The “D’oh!” (/doʊʔ/ DOH) segment to “Configuring it”

Notice theprivate_keyproperty in the pipeline YAML file. That is not good. You should not have keys, password and the like in files that commit to a repository. That is accepted knowledge. Why did I do it then? Because this is a hobby project. I’m the only one with access to the key and the repository. So it’s okay for now. But only for now. On the positive side. Then I have something to look into for an upcoming blog-post. Because, believe me! I have noted that this is something I have to look at.

Pushing the pipeline to ConcourseCI

Fly. Yeah let’s fly. As in hit the skies with the ConcourseCI cli AKA fly. Fly is a very integral part of ConcourseCI. As pointed out earlier in this blog post in the ConcourseCI - yeah what about it? section. When you feel confident and ready to throw your pipeline YAML file at the ConcourseCI backend do it like they do it where they do it, like this:

Install fly.
1. Go to your ConcourseCI backend root landing page. You’ll find the below.
2. Download the fly client for your OS.
3. Put it in whatever folder.
4. Make sure to put the folder in your OS environment PATH variable. You’ll know you have done it correctly when you can open up a new cmd/bash window and execute fly --version and get back a non-erring result (a version number).
Register the ConcourseCI backend.
1. fly --target "NAME_TO_GIVE_THE_TARGET" login --concourse-url http://"DOMAIN_OR_IP":"PORT"
Login to the ConcourseCI backend.
1. fly -t "THE_NAME_YOU_GAVE_THE_TARGET" login. You will be prompted for a password and a username. This is the one you specified when installing the ConcourseCI backend.
Push the pipeline.
1. fly -t "THE_NAME_YOU_GAVE_THE_TARGET" set-pipeline --config /PATH_TO_YOUR/pipeline.yml --pipeline "THE_NAME_TO_GIVE_YOUR_PIPELINE" --var qgit_pass="PASSWORD_TO_GIT
Notice the--var qgit_pass=****part of the fly set-pipeline cmdline. This is a great feature of ConcourseCI, that helps you avoid having passwords in your YAML file. So please use it .

Pro-tip! > use the\character to escape special characters in e.g. a password.

If you have followed along with vigour and a bright smile on your face. You will be able see something similar to the below on your ConcourseCI backend.

Specifics about the resources used in the ConcourseCI pipeline.

semver

Do remember to create a branch. Allow me to suggest you to name it “version” or something similar. As this will store the release version of the ‘thing’ you build.
- If you don’t you will get a > fatal: Remote branch version not found in upstream origin type error.
If you have a > SSL certificate problem: unable to get local issuer certificate error checking for new versions: exit status 128 type problem you should likely look into:
- that the entire certificate chain can be downloaded. Use the openssl cmdline tool to troubleshoot that. A laopenssl s_client -connect "troubling_domain":443 -servername "troubling_domain" -showcerts
- that you don’t have a LetsEncrypt certificate that was generated in a way that malformed it. Sorry I can’t be more specific as of now.

git

Use a proper certificate and ensure that the entire chain can be verified and downloaded. Use the same tips mentioned in the semver section above.

telegram

Getting the chat id. The easiest way I found was to:
- Create the bot by adding @BotFather to your Telegram CLI.
- Follow the instructions in order to create your bot.
- Send a message to your bot. Just some random crap. It doesn’t matter. this step is important! if you don’t execute it the response from the API endpoint below will just be an empty array/[].
- Execute https://api.telegram.org/bot'API-TOKEN'/getUpdates where the ‘API-TOKEN’ part is the token of your bot you just created via BotFather.
- In the JSON returned you will see a chat id in the chat section. That is the one you want to specify in the chat_id property in your pipeline YAML.

Learning outcomes. School’s out.

Eat your breakfast. Everyday. Yes! Everyday. Ooooops sorry. This is not some freakin’ paleo blog. Then again. We can all use a foodie tip now and then.
Seriously. Read the documentation on the software products you are trying to learn. Here is the good sources that helped me during my travels through the world of ConcourseCI.
1. https://github.com/concourse/docker-image-resource
2. https://concourse-ci.org/docs.html
3. https://github.com/concourse/git-resource
4. https://github.com/mrsixw/concourse-rsync-resource
5. https://github.com/w32blaster/telegram-notification-resource
6. https://wiki.qnap.com/wiki/SSH:_How_To_Set_Up_Authorized_Keys for connecting over SSH with Rsync.
7. https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted#668049 helped me in finding out why the files could not be written to the destination folder. It was a permissions issue. Owner had to be the user running the rsync job.

Thank you for reading along. Also a thank you to the kind people that helped me along my journey of getting this to work. They are:

Alex Suraci. One of the initial creators of ConcourseCI.
Christopher Brown. One of the initial creators of ConcourseCI.
Abraham Cabrera. A friendly ConcourseCI user on the ConcourseCI Discord need-help channel.
Morten Frydensberg. He’s a DevOps Engineer at Danske Spil and was so kind to listen to, both my clever and stupid ideas, as well as to troubleshoot the ConcourseCI worker container issue with me.

Over and out

Publishing the blog with ConcourseCI

ConcourseCI - yeah what about it?

ConcourseCI, let’s get at it.

What I wanted to accomplish with ConcourseCI

Installing it

The nitty gritty details of the conntrack issue (feel free to jump this)

The “D’oh!” (/doʊʔ/ DOH) segment to “Installing it”

Configuring it

The ConcourseCI pipeline YAML file

The “D’oh!” (/doʊʔ/ DOH) segment to “Configuring it”

Pushing the pipeline to ConcourseCI

Specifics about the resources used in the ConcourseCI pipeline.

Learning outcomes. School’s out.

Bengtsson Driven Development

Error

ConcourseCI - yeah what about it?

ConcourseCI, let’s get at it.

What I wanted to accomplish with ConcourseCI

Installing it

The nitty gritty details of the conntrack issue (feel free to jump this)

The “D’oh!” (/doʊʔ/ DOH) segment to “Installing it”

Configuring it

The ConcourseCI pipeline YAML file

The “D’oh!” (/doʊʔ/ DOH) segment to “Configuring it”

Pushing the pipeline to ConcourseCI

Specifics about the resources used in the ConcourseCI pipeline.

Learning outcomes. School’s out.

Templates:

Error