#cicd #coding #jenkins #nomad

You may ask why I spend time to setup a good old Jenkins while everyone else seems to jump on some newer CI systems (GitLab, Forgejo, etc.)? Well, as said, it's still good old Jenkins and I assume it will be around for some more time.

To run Jenkins agents on a Nomad cluster I followed Ola Ogunseghas instructions and example code here:

• https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57
• https://github.com/GastroGee/jenkins-nomad/blob/main/jenkins-controller/nomad.yaml

It involves installing the Nomad plugin for Jenkins and configuring this “Nomad cloud” (that's how the integration is called in Jenkins) with a template for the Jenkins agents.

Obviously, I wanted to integrate Jenkins with my Git repos. The most straight forward way seemed to use post-receive hooks to nudge Jenkins on every push. This has worked fabulously so far.

Even though the runners are spawned as Nomad jobs, I still wanted to run other Docker containers in the pipeline. This is where it got very confusing for me, because of the different Docker plugins for Jenkins. Most notably, there exist at least these two plugins:

• docker-workflow, runs the Docker container from the Jenkins agent
• docker-plugin, runs Jenkins agents as Docker containers

I decided to go with the former “docker-workflow” plugin because I already deployed the Jenkins agents as Nomad jobs. docker-workflow can run arbitrary containers from any Docker image, whereas the docker-plugin needs to be based on the Jenkins inbound-agent image to be able to connect to the Jenkins server.

I wanted the containers that are launched from Jenkins to be contained in a another users namespace. There exists the option to rebuild the inbound-agent with user-supplied attributes for uid/gid, but since I wanted to modify the image anyways I simply forked and built my own agent images, mostly inspired by the example for running Docker inside the agent.

At this stage, my tooling was sophisticated enough I could run a simple gitleaks container on each push to scan for secrets. I'm always afraid to publish secrets accidentally (it happened to me before).

Furthermore, I needed to establish some kind of build process, so I can also build images with Jenkins and directly push them to my local image registry. After some reading, I discarded the thought to use Kaniko because it still requires nodes in the native architecture of the respective build for multi-architecture builds.

Therefore, I followed RedHat best-practices to integrate Buildah, which is also the tool I use to build multi-arch container images locally on my laptop.

If you are interested in more example code, here the link to some of the key components:

• jenkins.yaml.tmpl.html Jenkins infrastructure code
• jenkins.nomad.html Jenkins nomad job
• docker-agent Modified Jenkins inbound-agent with Docker and Buildah

I'm not finished with playing around, because I still haven't figured out some things fully yet (e.g., how to properly use jlink in multi-arch builds) and because I'm also curious how other members of the community use Jenkins on Nomad.

You might ask why? I got asked about that stuff by my friendly office colleague Samuel. By the way, without him I would have never installed/used Nixos in the first place. Good fella, go checkout his blog.

#packaging #coding #nix #haskell

The goal was to package oama. Oama helps you to renew OAuth2 tokens from the command line. To be honest, I haven't checked the details on how it works yet, I was only interested in packaging it for Nix.

I was reading a bit on how it can be used with my favorite msmtp client.

Then I read some of the Nix documentation on how to build and use Haskell packages. They describe how to quickly bootstrap development environments (i.e., a Nix shell) using the handy cabal2nix utility.

The first step for packaging was also to convert the cabal description of the package into a valid Nix expression which already proved useful for development purposes.

git clone https://github.com/pdobsan/oama.git oama.git
cd oama.git
cabal2nix . > oama.nix

Create default.nix:

# Retrieve nixpkgs impurely from NIX_PATH for now, you can pin it instead, of course.
{ pkgs ? import <nixpkgs> {} }:

# use the nixpkgs default haskell package set
pkgs.haskellPackages.callPackage ./oama.nix { }

If you have a look at oama.nix it contains a lot of references to Haskell packages (.e.g, haskellPackages.aeson). The callPackage takes care of the name resolution in this case. haskellPackages.mkDerivation is a wrapper around stdenv.mkDerivation (see last post building a C project), but works without the standard Haskell build tool cabal-install.

To build oama and quickly run it in a shell:

nix-shell -p cabal-install
nix-build default.nix
./result/bin/oama

I did not look further into shellFor at this point, because I was interested in creating a Nix package that I can install globally.

For this, I took the “stdenv template” from the last post and merged 😻 it with the oama.nix output from cabal2nix. The result:

let
  pkgs = import <nixpkgs> { };
  hpkgs = pkgs.haskellPackages;
in
  hpkgs.mkDerivation {
    pname = "oama";
    version = "0.10.1";
    src = pkgs.fetchgit {
      url = "https://github.com/pdobsan/oama.git";
      rev = "refs/tags/0.10.1";
      hash = "sha256-mQBlCrF9rvFOfSOxhMi6JUKDJocFmO4Hhc3Zy7AqiXk=";
    };

    isLibrary = true;
    isExecutable = true;
    libraryHaskellDepends = with hpkgs; [
      aeson base bytestring containers directory hsyslog http-conduit
      network-uri optparse-applicative pretty-simple process string-qq
      strings text time twain unix utf8-string warp yaml
    ];
    executableHaskellDepends = with hpkgs; [
      aeson base bytestring containers directory hsyslog http-conduit
      network-uri optparse-applicative pretty-simple process string-qq
      strings text time twain unix utf8-string warp yaml
    ];

    mainProgram = "oama";
    license = pkgs.lib.licenses.bsd3;
  }

Essentially, I only had to use nix-prefetch-git to get the shasum of the git revision and make sure I use the pkgs and hpkgs (reference to pkgs.haskellPackages) correctly in the right places.

Afterwards, again, building and installing oama globally was a peace of cake 🧁:

nix-build oama.nix
nix-env -i -f oama.nix

Now I only have to figure out what to do with this tool 🤔 but this is left for another post.

After some reading on the Internet 📚, I finally understood the basics of building and installing a package from C sources using a Nix expression file (the .nix file in the following examples).

#packaging #coding #nix #diary #c

The expression file uses the standard build environment 🧰. Essentially, this environment abstracts the traditional Makefile based build proces (i.e., make && make install).

For learning purposes, I decided to package my hobby project, the text-based journaling program diary 📓.

Writing the Nix expression file

I built the expression file based on a an example from the Nix book.

The Nix expression file diary.nix to install the stable version of the diary program fetches the source by tag from the Git repository:

let
  pkgs = import <nixpkgs> { };
in
  pkgs.stdenv.mkDerivation {
    name = "diary";
    version = "v0.10";
    buildInputs = with pkgs; [
      ncurses
      curlFull
      lttng-ust
    ];

    src = pkgs.fetchgit {
      url = "https://git.in0rdr.ch/diary.git";
      rev = "refs/tags/v0.10";
      hash = "sha256-jDPfqUjf0ZETYqPxvEJ2/YFFjNsF7yvVQiThc5yMvjc=";
    };

    installPhase = ''
      install -d $out/bin
      install -d $out/share/man/man1
      install -m755 diary $out/bin/
      install -m644 man1/diary.1 $out/share/man/man1/
    '';
  }

The mkDerivation call does most of the work. There we define the package name and version, as well all of the dependencies required to build the project. In this case, the diary program depends on the curl and ncurses libraries as well as the lttng tracing library.

The source code src can be fetched conveniently from Git (using the fetchgit “builder”) or from other remote sources (see related “fetch” builtin functions).

To build a nightly release from the master branch, I only had to slightly tune the src of the unstable Nix expression:

    src = pkgs.fetchgit {
      url = "https://git.in0rdr.ch/diary.git";
      hash = "sha256-laKIVxfqJqBkB2d0F8tOOnpAXXRzHFfpjhAzZT8MX8k=";
    };

The sha256 hash can be obtained using nix-prefetch-git:

# get hash for the specific tag/release
nix-prefetch-git https://git.in0rdr.ch/diary.git --rev refs/tags/v0.10

# get hash for latest nightly build from master branch
nix-prefetch-git https://git.in0rdr.ch/diary.git

Finally, the installPhase needed a bit of tuning and has a small difference to what is commonly used in a Makefile on a traditional Linux distribution. But that was not too hard either, I simply had to translate my install target from the already existing Makefile.

The install target:

install: $(TARGET)
        install -m755 $(TARGET) $(BINDIR)/$(TARGET)
        install -d $(MANDIR)/man1
        install -m644 $(MAN1) $(MANDIR)/$(MAN1)

The corresponding Nix installPhase:

      install -d $out/bin
      install -d $out/share/man/man1
      install -m755 diary $out/bin/
      install -m644 man1/diary.1 $out/share/man/man1/

Very similar indeed. One point to note here is that the fixup phase will automatically install the man pages to the correct location.

Building and installing the C project

Once the expression file is written, building and installing the program is a peace of cake 🍰:

# Build stable package
nix-build nix/diary.nix

# Build latest nightly version
nix-build nix/diary-unstable.nix

To install the package from the expression file to the user environment:

# Install stable package
nix-env -i -f nix/diary.nix

# Install latest nightly version
nix-env -i -f nix/diary-unstable.nix

You can even verify the location of the man page by using manpath -q or query the installed package with nix-env -q.

Next steps

Now that we have drafted the basic expression file, we might also add it to the nixpkgs repository on GitHub. I haven't checked the contributor requirements in detail yet.

This app written in Python let's you easily read latest Swedish Teletext news. It's “a client for reading swedish text tv in the terminal”.

I slightly modified the code so it reads from Swiss 🇨🇭 Teletext source (SRF):

https://github.com/voidcase/txtv/pull/15

#coding #python #txtv

I could figure out the URL for the API by observing how the .gif pictures for the teletext.ch website are generated from the API while browsing through the txt pages:

• https://www.teletext.ch
• https://api.teletext.ch/channels/SRF1/pages/{num}

Unfortunately, I did not find a proper documentation for this API 😢.

Reading a bit of Javascript in the browser debugging console helped me to figure out the path structure of the https://api.teletext.ch API a bit better.

By the way, on my mobile I typically read https://m.txt.ch/SRF1 (but this one does not really work well for reading on the web browser / Laptop).

After playing a while with the API I realized, that there is not really an easy way to parse the structure of such a txt page from specific API fields.

I decided to only use the content field, because it already displays the Swiss German special characters and Umlauts quite well. Also, the header and other fields like commandRow where not really helpful to parse the content on a page (e.g., to substract title text):

curl -s https://api.teletext.ch/channels/SRF1/pages/130 | jq -r '.subpages[0].ep1Info.data.ep1Format.header'
ICAgICAgICAgICAgICBQMDEgICAgICAgICAgICAgICAgICAgICAgRA==

curl -s https://api.teletext.ch/channels/SRF1/pages/130 | jq -r '.subpages[0].ep1Info.data.ep1Format.header' | base64 -d
              P01                      D

In Teletext the pages generally follow a specific structure, most of the time.

Generally, there are some overview pages with links to subpages. The actual content can then be explored on these subpages.

The txtv CLI also roughly supports these two types of pages.

By default, txtv starts in the “interactive mode”, when not started with a specific argument (e.g., to read a specific page nr):

[andi@nixos:~]$ txtv
WHO: Kontakt nach Gaza gestört.......133
UNO bestätigt Gaza-Resolution........134
Maine: Mutmasslicher Schütze tot.....140
Frauen-Nati unterliegt Schweden......183
Lugano siegt bei ZSC Lions...........187
05:05 Gredig direkt/UT...............734
05:40 Hoch hinaus/UT.................735
06:25 News-Schlagzeilen 07:30 Wetterkanal KURZÜBERSICHT101
INLAND...............................104
SPORT................................180
AUSLAND..............................130
METEO................................500
WIRTSCHAFT...........................150
TV&RADIO.............................700

Very nice 😎 Not perfect, but still useful.

Now, for my patch I concentrated on displaying the information on two pages:

• INLAND 104
• AUSLAND 130

I also did some parsing regarding the “TV&RADIO” section but gave up on over-engineering that part after some time 🤓.

[andi@nixos:~]$ txtv 104
Sie folgt der Schwesterkommission....107
Aktionsplan Schweiz und Frankreich...108
Piste in Zermatt.....................109
Eine Stunde länger einkaufen.........110

[andi@nixos:~]$ txtv 130
WHO: Kein Kontakt zu Mitarbeitenden..133
UNO-Vollversammlung Gaza-Resolution..134
Reaktionen auf UNO-Resolution........135
Gaza: Kein Internet nach Angriffen...136
Weitere Hilfsgüter für Gazastreifen..137
Israel greift........................250
Hamas-Ziele an.......................138
Unesco fordert Schutz der Schulen.. 139MAINE/USA: Polizei findet Leiche140

As you can see from the examples above, the parsing rules still have some issues. This is due to the fact that I can only rely on a few basic assumptions on how the overview txt pages (e.g., 104 and 130) are structured. For instance, the current parsing is flawed when a page title includes a three digit number. For example “Israel greift 250 Hamas-Ziele an”:

Israel greift........................250
Hamas-Ziele an.......................138

The problem here is that 250 is interpreted as a page number, where it should not.

Unfortunately, I'm not aware of a solution for this, except for finding a smarter parsing rule:

# Find all three digit numbers, most probably these are page numbers
page_nrs = re.findall(r'\s(\d{3})*[-\/]*(\d{3})([^\d]|$)', stories)
all_page_nrs = []

for p in page_nrs:
    try:
        n = int(p[0])
        all_page_nrs.append(n)
    except:
        pass
    try:
        n = int(p[1])
        all_page_nrs.append(n)
    except:
        pass

all_page_nrs = [str(p) for p in all_page_nrs]

I experimented with quite a few variations, but gave up after some time, because each has it's benefits and pitfalls, where the rule would help in some cases but perform worse in another edge case.

So, if you have any suggestions on how to solve that problem of “missing intent” while parsing basically “generic text” let me know. I heard that in this age of AI and machine learning we should be able to do better, shouldn't we?

The same problem also applies when parsing the titles and subtitles on the overview pages. The txtv ls command is my favorite command. It simply lists all the subpages of the pages 104 (INLAND) and 130 (AUSLAND):

But here, as well, my parsing rule is flawed when it comes to actual page titles containing more than a few capital letters in a row (3-4 capital chars are typically used for abbreviations):

if self.num == 100 or self.num == 700:
    # Remove actual titles
    stories = re.sub("Jetzt auf SRF 1", "", stories)
    stories = re.sub("JETZT AUF SRF 1", "", stories)
    stories = re.sub("TELETEXT SRF 1", "", stories)
else:
    # Remove all uppercase subtitles. There can be multiple
    # subtitles on a page (subtitle, stories, subtitle, stories, etc)
    stories = regex.sub(r'[\p{Lu}\s-]{9,}[\s:]', '', self.content)

So in generally I can be quite happy with the outcome. It gives me a fast and smart way to quickly read the news on any terminal enabled device with Internet access 🤓. Also, the parsing rules on the individual detail/content/sub pages to extract the category and date work flawlessly (except maybe for the more special pages/categories like SPORT and TV&RADIO).

For future improvements I'm right now thinking to also use the links of the API output for extracting the set of real page numbers, which would at least help me with the first problem presented in this article (the titles that contain three digits):

curl -s https://api.teletext.ch/channels/SRF1/pages/130 | jq -r '.subpages[0].ep1Info.links'

For any page without links, I could simply apply the “content page” rendering which has shown to work pretty well already.

So far, I'm very happy with how things work and can do most of my tasks as on any other Linux distribution. Some tasks, however, require some more reading and thinking.

Today I wanted to jot down a quick how-to on testing an addition to the Perl packages in the Nix Packages collection (Nixpkgs).

#coding #perl #nixos

This how-to is based on:

• https://nixos.wiki/wiki/Perl#Adding_something_from_CPAN_to_nixpkgs
• https://dataswamp.org/~solene/2021-09-18-nix-cpan-pip.html#_Perl
• https://nixos.org/manual/nixpkgs/stable/#how-to-develop
• https://nixos.org/manual/nixpkgs/stable/#submitting-changes-making-patches

First, download nix-generate-from-cpan by adding it to /etc/nixos/configuration.nix and running sudo nixos-rebuild switch.

I won't go into the details on the first step here, because this is a task you should already be familiar with after having spent the first few hours configuring your NixOS installation.

The next steps involve patching the <nixpkgs> “collection” (which is part of the default nixpkgs “channel”) .

git clone --depth=1 https://github.com/NixOS/nixpkgs nixpkgs.git
cd nixpkgs.git

Then you create the patch for the pkgs/top-level/perl-packages.nix file as follows:

nix-generate-from-cpan Cal::DAV
attribute name: CalDAV
module: Cal::DAV
version: 0.6
package: Cal-DAV-0.6.tar.gz (Cal-DAV-0.6, CalDAV)
path: authors/id/S/SI/SIMONW
downloaded to: /home/andi/.cpanplus/authors/id/S/SI/SIMONW/Cal-DAV-0.6.tar.gz
sha-256: e436ecb61db9a2fa3ba4b2111149dfd31bdf7aa6acbc12cc49584e50db63e79b
unpacked to: /home/andi/.cpanplus/5.36.0/build/CM_4_0IbZK/Cal-DAV-0.6
runtime deps: DataICal HTTPDAV LWP
build deps: 
description: A CalDAV client
license: perl_5
RSS feed: https://metacpan.org/feed/distribution/Cal-DAV
===
  CalDAV = buildPerlPackage {
    pname = "Cal-DAV";
    version = "0.6";
    src = fetchurl {
      url = "mirror://cpan/authors/id/S/SI/SIMONW/Cal-DAV-0.6.tar.gz";
      sha256 = "e436ecb61db9a2fa3ba4b2111149dfd31bdf7aa6acbc12cc49584e50db63e79b";
    };
    propagatedBuildInputs = [ DataICal HTTPDAV LWP ];
    meta = {
      description = "A CalDAV client";
      license = with lib.licenses; [ artistic1 gpl1Plus ];
    };
  };

Add the expression in alphabetical order to pkgs/top-level/perl-packages.nix. Only copy/paste the “Nix expression” after the === characters into the file.

To test the patch, spin-up a nix-shell which includes the new Perl module. Create the default.nix expression file in any working directory:

cat <<EOF > default.nix 
with (import <nixpkgs> {});
let
  CalDAV = perlPackages.buildPerlPackage {
    pname = "Cal-DAV";
    version = "0.6";
    src = fetchurl {
      url = "mirror://cpan/authors/id/S/SI/SIMONW/Cal-DAV-0.6.tar.gz";
      sha256 = "e436ecb61db9a2fa3ba4b2111149dfd31bdf7aa6acbc12cc49584e50db63e79b";
    };
    propagatedBuildInputs = [ perlPackages.DataICal perlPackages.HTTPDAV perlPackages.LWP ];
    meta = {
      description = "A CalDAV client";
      license = with lib.licenses; [ artistic1 gpl1Plus ];
    };
  };

in
mkShell {
  buildInputs = [ CalDAV perl ];
  # putting perl here is only required when not using NixOS, this tell you want Nix perl binary
}
EOF

The expression includes the result of the previous nix-generate-from-cpan Cal::DAV with slight modifications.

Notice that propagatedBuildInputs needs to refer to already installed packages prefixed by perlPackages.

More information on how to develop with nix-shell.

Solènes article nicely shows how the let expression can also include an entirely new block of code to define a variable. In this example, the let expression uses the buildPerlPackage function, which is a “builder” for building Perl packages (as the name indicates).

With that said, you can launch the development shell to test any Perl script with the newly added Perl library as follows:

nix-shell

If you don't want to create a “descriptive” shell expression, you can also launch the shell more in an ad-hoc fashion using the following command:

nix-shell -p perlPackages.CalDAV perl -I nixpkgs=~/Downloads/nixpkgs.git

See also the example for Perl in the nix-shell Manual.

If you are happy with how the module works in the nix-shell, you can add it to your system/profile (using nix-env or nixos-rebuild) and even contribute it to the upstream <nixpkgs> repository.

This post mostly reminds myself of all the new terminology I learned the past few days.

• Project site
• Demo instance
• Code

#coding #sports #myheats

I took advantage of the CSS flexbox layout to improve the design on mobile devices. The main idea was to reduce the table size to a minimum on small screen sizes. The individual cells of the table are shown as small boxes, with labels to indicate the content type.

On larger displays, for instance, a big screen at the finish line of a sport event/run, the table is displayed in a traditional fashion.

Here some more screenshots of the admin section with the new design.

The live-updating leaderboard:

The scoring section where judges can enter the scores. In this section, I found it important to indicate the loading status whenever the score is changed and to make the input field big enough for usage on small smartphone screens:

Friends told me that similar software has a feature to “lock” the scores once the scoring is done. Currently, there exists no such feature in MyHeats, but that's a good idea for my next hacking sess.

The section where event admins can create heats and manage the startlists for each run:

The administration of the athletes:

Athletes are not required to have a lastname. In this way, the leaderboard can equally well be used to score entire teams.

And last but not least, if you are reading until down here and you are interested in the technical implementation and the arrangement of the data in the database, here the database schema:

Interested in test-driving the scoring backend for judges and event admins? Drop me a line (see footer).

• 🚀 Demo instance: https://myheats.in0rdr.ch
• 🧬 Code: https://code.in0rdr.ch/myheats

Plan to work on some features the next few days: * Add/remove athletes (bulk) * Improve UI

#coding #sports #myheats

For more background on the app, the idea and the problem(s) it could solve have a look at the original post Live heats, Scoring and Leaderboard for Sport Events (March 11, 2023).

edit 2023-04-04: Bug fixed in v0.2, thanks @JonnyBurger and @PatrickH!

oh.. and, I might need your 🥺 help with a little 🐛 bug I encounter.

⁉️ Anyone knows where this flickering comes from? Reproduce as follows:

1. Add one heats to display
2. Remove the heat again
3. Wait for the screen to update. Can you see that flickering?

How can I fix that? I'm pretty new in that JavaScript business. If you know the fix, please shoot me a mail, ping me (see footer).

The bug should be in Leaderboard.js somewhere.. Thanks for any hints!

#coding #sports #myheats

I detected the following deficiencies which could be improved with a more digital approach (digital/mobile from the start):

• Mistakes can occur while copying ratings and numbers from paper into Excel
• Sorting each Excel list (for every competition and run) manually is error prone, especially, with multiple competitions going on simultaneously (slopestyle ski or snowboard, formation skiing, men, women, several runs, etc.)
• The results need to be printed on paper and published on the website (PDF)
• Formatting and printing these sheets (for each competition) takes quite some time and the result is not always consistent (in terms of formatting)
• There are no live results. This is also not convenient for the athletes that want to know right after the run how they scored and how they rank in the competition. Ideally, the athletes get to know their scores right after they cross the finish line.
• OneDrive syncing did not always work reliably (depending on the end users devices) and mailing the sheets was a pain

I was looking into some existing solutions for systems to process the data, such as liveheats. However, since most of these systems are finished products and there is no open source code available that I know of, I decided to quickly draft my own version of a live leaderboard for these kind of sport competitions.

🚀 Demo instance: https://myheats.p0c.ch

🧬 Code: https://code.in0rdr.ch/myheats

Right now, there is only the leaderboard which automatically updates as soon as new data arrives in Supabase. During the process of creating this first draft, I became a huge fan of Supabase because it works really well. It is the open source Firebase alternative with Postgres database in the backend. I can highly recommend it for your next project, because it is extremely easy to get started with.

For the purpose of the leaderboard, I was especially interested in the realtime aspects of Supabase. This allows me to update the table as soon as new data is entered in the backend (by the judges of the comptition).

I'm planning to update the code with some more features soonish, among others:

• Authenticaiton for the experts that perform the ranking and rating
• A very simple user-interface or view for the experts to edit the score for a single athlete of a heat or run
• Better import/export functionality for start list and rankings

Stay tuned. If you have some input or tips feel free to drop me a line (see footer).