When you are lucky good. A story on a rotten character. That trashed .netrc and therefore ConcourseCI resources.

16 Oct 2018 in Devops / Troubleshooting on Concourseci, .netrc, Git, Special-characters

This is a journey into the world of troubleshooting, pulling out ones own hair and coming out wiser and happier.

So I had this issue:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
Cloning into '/tmp/semver-git-repo'...
fatal: could not read Username for 'https://bengtssondd.it': No such device or address
error checking for new versions: exit status 128`

This happened when the semver (semantic versioning) resource of the CI product named ConcourseCI tried to poll my private Git backend. It was rather frustrating, as it meant my tirade into setting up a proper build pipeline for the HealOps project went into a total halt. Furthermore, the same error occurred for the git ConcourseCI resource.

Then what is a man to do? Find the culprit of course. Then smack it in the face….like…kapow!

So I tried

• Downgradring to v3.10 of ConourseCI where it used to work. It still didn’t.
• Thinking about what I had changed in my ConcourseCI setup.
  • I had “upgraded” the docker-compose file to v3.2.
  • Updated the Postgres SQL database backend used by ConcourseCI to v10.5.
• Then I tried setting up ConcourseCI, the HealOps build pipelin on another Macbook I have. And no error. It works. But why in the world.
  • Then I considered what the difference between the two Macbooks is. The MacBook where it works do have a newer version of Git installed. But could that really be it? Does the semver and git resources really use the local version of Git? What is going on here? It would be great to get a clearer view on this issue.
• I followed that line of thought and updated Git on the non-working Mac. I even did a full Docker restart on MacOS. And a full re-install of ConcourseCI (I also removed the Postgres docker volume). The issue remained.
• Then I saw someone mentioning the fly hijack command that ConcourseCI have. I went ahead and “hijacked” the worker container in which the semver resource is running. I executed git clone https://bengtssondd.it/git/healops.git to test git and was prompted for a username and password. When entering those two correctly the git clone command executed without problems. The issue still remained though.

Hmmm. Now what. What do you do when you are in the center of the mist are kinda lost. You poke around. Try out new routes. Wave your arms. Find your smartphone flashlight.

So I poked around while in fly hijack mode. Did a little ls -la in various places. Aaaand what was that. In the root of the container filesystem there was a .gitconfig file. Let’s cat .gitconfig that. Hmmmm nothing in there of interest. Oh my. There is also a .netrc file in the. Let’s hit that with a cat .netrc as well. Aaaand what do we have in there my friend?

We had.

default login myusername password \!MyPassword((88))

Immediately I noticed the leading \. It shouldn’t be there as it is not a part of the password. It is there in order to act as a escape character to the ! character. Necessary when executing the fly -t myConcoursePipeline set-pipeline --config ./pipeline.yml -p PipelineName --var myusername=\!MyPassword((88)). If not the ! character will be interpreted as a special-character and not as a regular character that should be a part of the value to the myusername variable. As soon as I saw that I removed the \ character via Vi. Then tried my git clone https://bengtssondd.it/git/healops.git command again. This time with no prompts. I think we are there. A nice bubbly feeling started to ooze in my chest . I looked up .netrc and I was further re-assured in this being the culprit. .netrc is a part of the Linux auto-login system. It is used for providing login info automatically as the name implies. I tested by updating the HealOps pipeline yml file » comitted it to ConcourseCI with the fly -t myConcoursePipeline set-pipeline.... command. Waiting … tic … toc … tic … toc .

Ta-da freaking laaaaaa. It worked. That was it. A special-character f** up my life. However, nice that I was able to half troubleshoot myself en route to the culprit, half stumble over it.

Now I just have some failing Pester tests when HealOps is building. More on that in a later post.

For now, onto filling out a bug report on the Concourse semver and git resource GitHub pages. I think special characters should be supported. The bug report for reference.

Thank you for reading along.

Over and out