Deleting old Rundeck logs / executions

Rundeck is an open source automation service with a web console, command line tools and a WebAPI. It lets you easily run automation tasks across a set of nodes.

It’s quite convenient to automate most of the tedious, repetitive maintenance and housekeeping tasks with Rundeck. Unfortunately if you use Rundeck for a long time, you might experience a huge amount of log files. As it turns out, Rundeck does not automatically clean up old log files. In my case, I had a task running like every 30 seconds. This lead to more than 150.000 executions and a few gigabytes of log files in a few days.

This behaviour isn’t new to the maintainers of Rundeck and there is / was an active discussion about this topic. The issue is concluded with the statement

the workaround is to use the rd executions deletebulk command.

Means: Build it your own and use the fancy Rundeck CLI to cleanup the old logs. Wait a sec… I could automate this and use Rundeck to clean Rundeck. This would be some kind of Rundeckception.

Fortunately Alex Simenduev alias shamil already dealt with this issue and published a suitable Gist:

#!/bin/bash -e

# export RD_URL=https://rundeck.example.com RD_USER=admin RD_PASSWORD=admin RD_HTTP_TIMEOUT=300

# make sure rd & jq commands are in the PATH
which -- rd jq >/dev/null

del_executions() {
    local project=$1

    while true; do
        rd executions deletebulk -y -m ${RD_OPTION_BATCH:-20} --older ${RD_OPTION_OLDER_THAN:-7d} -p $project || break
        sleep 1s
    done
}

del_executions $1

exit 0

I made some minor adaptions to his original version, as the original loop had to be cancelled manually. The script requires the Rundeck CLI and jq to be available on the system. You can use the following Dockerfile as a quickstart for cleaning up your old Rundeck logs:

FROM thofer/rundeck-cli

ENV RD_URL=https://rundeck.example.com \
    RD_USER=admin \
    RD_PASSWORD=admin \
    RD_HTTP_TIMEOUT=300

COPY clean.sh /tmp

RUN apt-get update -y && apt-get install -y jq
RUN chmod +x /tmp/clean.sh

ENTRYPOINT ["/tmp/clean.sh"]

Build it by putting both files (clean.sh and the Dockerfile) into the same directory:

docker build -t rdc
# rdc = rundeck-cleaner

And execute it afterwards by passing the respective ENVs with the targeted project name as the first argument:

docker run -it -e RD_URL=https://rundeck.unicorn.com -e RD_PASSWORD=Unicorn rdc Housekeeping
# wheres as „Housekeeping“ would be the project name (refer to the Rundeck admin interface)

Tesseract OCR: ParamsModel::Incomplete line

Using Tesseract OCR for some automation tasks, I stumbled upon the following issue:

Creating OCR Image #1/4 with maximally 97 lines from line 1 to 97
ParamsModel::Incomplete line
ParamsModel::Incomplete line
ParamsModel::Incomplete line
ParamsModel::Incomplete line
ParamsModel::Incomplete line ConvNL

As it turns out, this is caused by using a wrong version of tesseract models / „Tessdata“. You can find the official „Tessdata“ files on GitHub. As I installed Tesseract on macOS via Homebrew, I ended up with version 3.05.01 of Tesseract. Therefore I had to use an older version of the trained models. As stated in the project’s README, you can find those via the 3.04.00 tag (README says branch, but that is wrong):

Get language data files for Tesseract 3.04 or 3.05 from the 3.04 tree.

E.g. for the german model, you would end up with the following download link:

https://github.com/tesseract-ocr/tessdata/raw/3.04.00/deu.traineddata

Cordova: All flavors must now belong to a named flavor dimension.

Messing around with Cordova I stumbled upon the following exception when building the app for Android:

A problem occurred configuring root project 'android'.
> All flavors must now belong to a named flavor dimension. Learn more at https://d.android.com/r/tools/flavorDimensions-missing-error-message.html

The almighty folks over at Stackoverflow suggest to update the build.gradle and add a flavorDimensions parameter into the android section. As this smells like a dirty workaround for an issue that should (as always) not be happening in the first place, I found some deeper digging into this topic from Dave Alden. To sum it up: cordova-android@6.4.0 requires Gradle v4, but cordova-plugin-crosswalk-webview is incompatible with Gradle 4.

This leaves you with 3,5 options:

  • Upgrade cordova-android to 7+: This might mess up all your other plugins as they did some breaking changes (that’s why I still had to stick to 6.4.0). Quite a lot of Cordova plugins are not yet compatible with Cordova v7+ as of today.
  • Downgrade cordova-android even further: This has the risk of some other side-effects, missing stability and security updates, etc.
  • Downgrade Gradle to v3: This might lead to incompatibilities with e.g. cordova-plugin-crosswalk-webview
  • Abandon Cordova altogether and switch to React Native, Xamarin or Weex… but yeah.

Further references

CocoaPods can’t reach GitHub

Running CocoaPods on “old” versions of OS X / macOS might lead to some weird behaviour – namely CocoaPods telling you that GitHub might be down:

$ pod repo update --verbose

Updating spec repo `master`
[!] Failed to connect to GitHub to update the CocoaPods/Specs specs repo - Please check if you are offline, or that GitHub is down

/Library/Ruby/Gems/2.0.0/gems/cocoapods-core-1.3.1/lib/cocoapods-core/github.rb:105:in `rescue in modified_since_commit'
[...]

The error message does not only sound insane, it is actually completely misleading. The issue seems to be caused by a failing request to GitHub. Originating from a failed TLS handshake (as it happens quite often with the older Ruby 2.0.0 version). More recent versions of Ruby don’t have a problem with communicating via SSL. Using a more recent Ruby version does in fact circumvent the issue:

brew install ruby
sudo gem install cocoapods

Depending on your system, you might need to forcefully “overwrite” the Ruby installation using brew link:

brew link --overwrite ruby

Further references

OpenShift 3.9 missing Webconsole

Getting started with OpenShift 3.9 differs from the prior versions. The OpenShift webconsole, which was provided by default in the past, seems to be no longer shipped and enabled by default. Trying to access via https://oc.example.com:8443 reveals the following (unhelpful) message:

missing service (service “webconsole” not found)
missing route (service “webconsole” not found)

Digging through GitHub issues reveals, that this behavior is intentional and the users have to deploy the Webconsole manually from now (at least if not relying on the official Ansible automation). The required deployment / route / service configurations are provided in the official OpenShift Origin repository. Running the following commands gets you up and running with a fresh webconsole under OpenShift Origin 3.9:

$ git clone https://github.com/openshift/origin
$ echo "Optionally checkout the specific version you are trying to get running"
$ oc login -u system:admin
$ oc create namespace openshift-web-console
$ # Customize install/origin-web-console/console-config.yaml before running the following command
$ # Replace 127.0.0.1 with you own IP / domain if you are not running locally
$ oc process -f install/origin-web-console/console-template.yaml -p "API_SERVER_CONFIG=$(cat install/origin-web-console/console-config.yaml)" | oc apply -n openshift-web-console -f -

Fortunately the uggly error message will most probably disappear in subsequent releases. There is an open PR dealing with it: Bug 1538006 – Improve error page when console not installed

Further references

Prometheus 2: Replacing count_scalar (unknown function)

Prometheus 2 has been released a few weeks ago and did affect quite some parts. One of which is replacing existing functions thus breaking existing Grafana dashboards in some circumstances:

parse error at char 96: unknown function with name "count_scalar"

Scanning through Pull Requests dealing with this change, it seems as if scalar(count()) is a suitable replacement. See e.g. what Percona did in one of their Pull Requests: PMM-1676 Parse error in Grafana dashboards due to removed count_scalar

Just replace your broken dashboard queries using count_scalar (don’t get confused with the parens):

count_scalar(...) -> scalar(count(...))

Further references

How to build you own Open GApps package

The Open GApps project provides a convenient way to get up-to-date Google App packages (most often used in combination with custom ROMs). Unfortunately they do not always offer the most recent versions and it takes some time until new Android releases are reflected on the official Open GApps project website. As of this writing, Android 8.1 is the most recent Android version and is not yet in the portfolio of the Open GApps project.

Fortunately they publish their automation and all necessary assets on GitHub. Therefore building your package from their sources is pretty feasible. The following guide is done for macOS, but should be quite similar to most Linux distributions (hint: you can use the beevelop/android Docker image to save some time):

# Install lzip via brew (Open GApps depends on it)
brew install lzip
# Clone the main repository
git clone git@github.com:opengapps/opengapps.git
# Download the sources for your targeted architecture (arm64 in my case)
# Downloading and „uncompacting“ the repositories takes quite some time
# Get a coffee or two in the meantime
./download_sources.sh --shallow arm64

# The final step is building the package itself
# This also does take quite some time
# especially depending on your CPU power (due to compression stuff, etc.)
make arm64-27-stock

The following script might help you getting started by using the Docker image mentioned above. Just run the following commands inside the Docker container (e.g. docker run -it --rm beevelop/android):

apt install build-essential lzip git zip
# SSH-Key on the machine is required and has to be added to your GitHub account
git clone git@github.com:opengapps/opengapps.git
./download_sources.sh --shallow arm64
make arm64-27-stock
# The command should great you with:
# SUCCESS: Built Open GApps variation stock with API 27 level for arm64 as [...path...]

Afterwards you can transfer the resulting zip file from Docker container to your host machine using docker cp:

docker cp practical_wilson:/root/opengapps/out/open_gapps-arm64-8.1-stock-20180203-UNOFFICIAL.zip .
# from there on scp it to your local computer and put it on your gorgeous mobile phone

SSH: Automatic notifications on login (Pushover)

The following snippet enables you to automatically trigger Pushover notifications for every SSH login on your server / NAS / desktop:

API_TOKEN=abcdefg1234hijklmno567890pqrstuv
API_USER=vutsrqp098765onmlkjih4321gfedcba
if [ -n "$SSH_CLIENT" ]; then
  USER_IP=`echo $SSH_CLIENT|awk '{print $1}'`
  TITLE="SSH: ${USER}@$(hostname -f) (${USER_IP})"
  TEXT="$(date)"

  curl -s \
  -F "token=$API_TOKEN" \
  -F "user=$API_USER" \
  -F "title=$TITLE" \
  -F "message=$TEXT" \
  -F "priority=0" \
  https://api.pushover.net/1/messages.json >/dev/null 2>&1
fi
Screenshot of Pushover notification
Just replace `API_TOKEN` and `API_USER` variables with your personal values. You can also customize the `priority` parameter to suite your needs. `1` would be high priority, whereas `2` requires you to acknowledge the respective notifications.

Put the above snippet into your server’s `/etc/ssh/sshrc` file to automatically inform you about every SSH login. Using `sshrc` is the „convenient“ solution and does not work, when the a user decides to override it e.g. with a `~/.ssh/rc` file. Refer to the following Stackoverflow discussion for further information and possible alternatives: https://askubuntu.com/questions/179889/how-do-i-set-up-an-email-alert-when-a-ssh-login-is-successful

Installing Delve on OS X / macOS fails

While trying to debug Go scripts with Visual Studio, I stumbled upon issues configuring Delve on macOS. Delve is a full featured debugging tool for the Go programming language and Visual Studio Code seems to make us of it. Unfortunately the installation might not be as straightforward as described in the official installation documentation. The Brew installation fails while / after generating certificates:

==> Installing delve from go-delve/delve
==> Downloading https://github.com/derekparker/delve/archive/v1.0.0-rc.2.tar.gz
Already downloaded: /Users/beematik/Library/Caches/Homebrew/delve-1.0.0-rc.2.tar.gz
security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain.
==> Generating dlv-cert
==> openssl req -new -newkey rsa:2048 -x509 -days 3650 -nodes -config dlv-cert.cfg -extensions codesign_reqext -batch -out dlv-cert.c
==> [SUDO] Installing dlv-cert as root
==> sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain dlv-cert.cer
Last 15 lines from /Users/[...]/Library/Logs/Homebrew/delve/02.sudo:
2017-12-31 11:52:35 +0100

sudo
security
add-trusted-cert
-d
-r
trustRoot
-k
/Library/Keychains/System.keychain
dlv-cert.cer


If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/go-delve/homebrew-delve/issues

These open issues may also help:
Upgrade to delve fails https://github.com/go-delve/homebrew-delve/issues/20

Whatever the reason of this issue might be, there is a suitable workaround available:

cd $HOME/Library/Caches/Homebrew
tar xf delve-*.gz
cd delve-*
sh scripts/gencert.sh
# Type in your sudo password
brew install go-delve/delve/delve

References

  • https://github.com/derekparker/delve/blob/master/Documentation/installation/osx/install.md
  • https://github.com/go-delve/homebrew-delve/issues/19#issuecomment-330442033

Affected System

  • OS X El Capitan 10.11.6
  • Homebrew 1.4.1
  • Delve 1.0.0-rc.2

Installing Python with PIP on Boot2Docker

Boot2Docker is quite awesome for getting started with Docker on Windows or OS X / macOS. Tools like „minishift“ do still use Boot2Docker to simplify their setup. Unfortunately Boot2Docker does not provide a solid package manager by default (like most Linux distributions do nowadays with apt, yum, etc.). But as Boot2Docker is based on Tiny Linux, it offers tce-load for some basic package management.

tce-load even provides a full-blown python package, what enables us to install Python with PIP on Boot2Docker. Running the following snippet gets you up and running:

tce-load -wi python
curl https://bootstrap.pypa.io/get-pip.py | sudo python -

Further references

  • List of installable packages for tce-load: http://distro.ibiblio.org/tinycorelinux/tcz_2x.html
  • Documentation of the tce-load command: http://wiki.tinycorelinux.net/wiki:tce-load