Using microk8s with system kubectl

I was already using kubectl with some plugins, and the nice completion already set up, to access remote Kubernetes clusters from my Ubuntu desktop. So when I decided to try microk8s, I wanted to keep the same convenience, without maintaining two configs, or having two different commands. It took a little bit of research to figure out how to make it work, so here it is.

First, find the admin password for your microk8s:

microk8s.kubectl config view | grep password

Now you can create a new context in kubectl, adding in your password on line 2:

kubectl config set-cluster microk8s --server= --certificate-authority=/var/snap/microk8s/current/certs/ca.crt

kubectl config set-credentials microk8s-admin --username=admin --password={password from before}

kubectl config set-context microk8s --cluster=microk8s --namespace=default --user=microk8s-admin

Now you can switch to the new context with kubectl config use-context microk8s and work as before. kubectl config get-contexts will list the other available contexts so you can switch back.


TIL: Putty “Server refused our key”

I’ve had this strange problem for about a year or more, where on my main Windows system, I can’t log in to local Linux servers using SSH Public Key authentication. I get this “Server refused our key” message. The same key files on a different computer work fine though.

Tonight I finally got annoyed enough to investigate deeper. Googling did not help me, so here’s a blog post so that it might help you. The final straw was installing the RancherOS ISO, which exclusively uses SSH keys, and not being able to get into it from my normal desktop. I suspect this is not the most common cause of “Server refused our key” errors, especially if you are getting them from multiple client to the same server. Look at the server’s SSH config in that case.

I use Pageant, and it has my usual keys loaded at boot time.

So I enabled debug logging in Putty and SSHed to my Linux desktop.

With that enabled, I was able to see this in the logs:

Event Log: Reading key file "C:/xampp/htdocs/cacti-1.0.6/plugins/weathermap/dev/Vagrant/.vagrant/machines/develop/virtualbox/private_key.ppk"
Event Log: Pageant is running. Requesting keys.
Event Log: Pageant has 4 SSH-2 keys
Event Log: Configured key file not in Pageant

So it turns out that at some point I had a Vagrant box running, and saved its private key into my DEFAULT putty profile. From there, it got saved into other profiles. Further, if putty had a specific key in this box, and it no longer exists, it never gets as far as the Pageant-supplied keys, and just gives up!

Removing that entry (at the bottom of the SSH auth section) has cured all my problems, and RancherOS is back on the menu!


Reusing DHCP config in ISC dhcpd

This is quite a niche thing, but since it took me ages to piece it together, here it is.

I run a PXE build environment for bare-metal install of new systems. I use PuppetLabs’ Razor to do the installation bit, and that works well. I do this for multiple environments, and I wanted to have a single server serve all of these related subnets (they come and go as projects complete). The single server acts as TFTP, DHCP, & Razor server for each subnet, as well as a NAT for external access, so it’s possible to use any IP addressing inside the build environment (except the network the razor box is on, of course).

In each subnet, the DHCP config is somewhat complicated. Aside from the normal DHCP stuff (IPs, gateway etc), the boot file to be used varies depending on the client-id of the host. EFI devices get a difference iPXE boot file from BIOS devices. Also, iPXE then makes its own DHCP request, which needs to be given a script instead. The script filename is different per subnet. The naive way to do all that results in an enormous tangle to update as projects change.

So my goals:

  1. Single copy of the boot file switchy config
  2. Simple to update/edit config for new subnets

For the switching stuff, I started with this Gist from robinsmidsrod, and chopped out the cases I don’t care about. That leaves me with this include file:

option 1;

if exists user-class and option user-class = "iPXE" {
    filename "bootstrap-";
elsif option arch = 00:06 {
    # EFI 32-bit
    filename "ipxe-x86.efi";
elsif option arch = 00:07 {
    # EFI 64-bit    
    filename "ipxe-x64.efi";
elsif option arch = 00:00 {
    # Legacy BIOS x86 mode
    filename "ipxe.pxe";

and scopes like:

# Network 1
subnet netmask {
   option routers;

   include "/etc/dhcp/ipxe-common-";

So that works, but has a lot of duplication! Being able to define parameters would be useful.

So that looks like this:

option bootstrap_file code 254 = string;

# Network 1
subnet netmask {
   option routers;

   option bootstrap_file "bootstrap-";

   include "/etc/dhcp/ipxe-common.conf";

and then:

if exists user-class and option user-class = "iPXE" {
    option bootfile-name = concat("",config-option bootstrap_file);

in the included file means that nothing needs to change in there anymore. I can just add a new scope in a few lines, drop in the appropriate TFTP file and be up and running.


Getting GNU Smalltalk to work on Ubuntu 16

I thought I’d play with Smalltalk. GNU Smalltalk is free, and also has the classic Smalltalk browser application. Except out of the box, it fails on Ubuntu 16 with:

Object: CFunctionDescriptor new: 1 "<0x7f1ffaacf690>" error: Invalid C call-out g_date_get_type

So here’s the packages you need to install so that gst-browser actually starts up on Xenial:

sudo apt install gnu-smalltalk-browser gnu-smalltalk gnu-smalltalk-doc libgtk2.0-dev libcairo2-dev libfontconfig1-dev libfreetype6-dev libpng-dev libpango1.0-dev libgdk-pixbuf2.0-dev

Now I can read the nice Adele Goldberg books I bought from thriftbooks, and play along 🙂


Rack diagrams with Weathermap

This is a response to a github issue for Weathermap. This way, everyone might benefit! (for those that don’t know, I write an open source tool to generate diagrams with live data from your monitoring system)


This is a demo I did years ago for the datacentre I was working in then. We were working on an asset register system, and I was hoping to be able to generate these maps from that data. You should be able to do the same kind of thing with racktables or OpenDCIM.

The key parts are:

  • A column of ‘notch’ nodes. All but the first one are positioned relative to the one above. These are a simple gif image.
  • The actual servers and equipment. Those are positioned relative to the notch nodes. We used a combination of rack-diagram images from Visio, and actual photos taken square-on to the front of racks. Pick a size for 1U, and that will dictate what 19 inches across is. For us, it was 400 pixels. Use that as a scale on all your icons. The vertical will fall into place, but set the Y scale on your icons to something large, so that it will never be the ‘important’ scaling dimension.
  • Status indicators. Those are positioned relative to the servers.

The original version also had some nice fonts defined, but distributing fonts is much more complicated so I changed the downloadable example to use the built-in fonts.


So first, we define a template node for how to draw a ‘notch’. This contains the icon, and the label, so they don’t need to be repeated.

NODE notch
    LABEL {node:this:name}
    LABELFONTCOLOR 255 255 255
    ICON 1u-notch.png

Then we define all the notches.

    TEMPLATE notch
    POSITION 150 100

    TEMPLATE notch
    POSITION U1 0 39

    TEMPLATE notch
    POSITION U2 0 39

    [etc etc]

You can generate this code in Excel or a scripting language very easily. Or just stick with my dimensions and use the template in the zip file.


Next comes the server. For a 1U server, things are simple. We just use relative positioning from the appropriate U notch. For larger servers we need to do a little maths: the node is positioned from its centre, so we need to know that a 2U server is 19 pixels down from the first U position it occupies.

NODE server2
    LABEL Web Server 3
    ICON 400 300 dell2u.png
    POSITION U6 225 19

NODE server3
    LABEL Web Server 3
    ICON 400 300 dell1u.png
    POSITION U9 225 0


Finally, if you need to show the current state from somewhere, you can easily add indicators:

NODE server3_state
    ICON 20 20 red-ball-64.png
    POSITION server3 -270 0

(obviously you would add some TARGET information to get the current state)

Note that everything is positioned relatively. This means you can move the whole rack by moving one thing (the U1 notch). The status indicator will follow the server if you move it, too.


I’ve zipped up an example map, some server images, and a blank 42U rack config file that can be used as a template for your own racks:


Modernising the Windows Console

My work laptop is stuck in Windows-land really, because of some required corporate applications. Lately I’ve been writing a fair bit of Python, and using the coloredlogs extension to the logging module to produce pretty logging from my code. Except on Windows, where the standard CMD prompt doesn’t understand ANSI colour codes, which is kind of a pain.

There are a few solutions that patch the default command prompt, to add back in ANSI support (DOS always had it available, way back when), but they all seemed a bit hacky.

So I switched to ConEmu, which provides a huge list of features, including ANSI colour support, tabs, all kinds of display modes, transparency, xterm-style cut&paste, and a canned list of contained applications, including CMD, Powershell, Chocolatey, Cygwin and that funny shell with all the environment set up that Visual Studio uses. Of course, you can also add your own.

Now I have a nicer-looking and nicer-acting console, and my Python logging colours work properly!

After playing with that for a while, I noticed the option to enable Clink inside ConEmu. Clink adds a large range of bash-style features to Cmd itself, like saved history, incremental history search and fancy completion, as well as simple things like being able to press Ctrl-V to paste, and readline editing (emacs keys to edit the input).

Projects & Hacking Toys

Making old joysticks from new ones…?

I have a reasonable collection of 80s home computers and consoles, many of which use the “standard Atari” joystick port. Those Atari CX40 joysticks, which is where this standard comes from, are particularly crappy though, so most gamers at that time would use something else with the same wiring, like a classic Competition Pro. You can get Competition Pros on ebay, but they are upwards of £30 – so what alternatives are there?

2015-09-29 16.10.59-sized

I have a Speedlink USB Competition Pro that I bought from Amazon last year (for much less than £30!) with all-microswitch switching but a USB port – so I opened it up. Inside it’s actually a not-very integrated circuit, which is good news for un-USBing it. There’s a little wiring harness for the microswitches, all going back to a tiny PCB where the autofire switch is. A new small wiring harness, a DE-9 socket and a 8-core shielded cable, and we’re in business, with a brand-new moulding and switches! (or the cable from a dead CX40 or similar). People like Digikey will sell you a 6ft ribbon cable with a DE-9 connector already attached at one end for a few pounds, so then it’s just the wiring.

Here’s some pictures of the insides, in case anyone else was wondering – just a small blob (presumably a PIC or similar) and the wiring connector, really:

2015-09-29 12.17.22

2015-09-29 12.17.41


From DVD ISO to mkv

I spent a while ripping DVDs to ISO files using DVD Decrypter a while ago, which allows me to use the Kodi (XBMC) menus to select DVDs to watch from anywhere in the house. However, lately I’ve been interested in Plex Media Server, which, although it’s a fork of XBMC, doesn’t support the ability to play DVD ISOs. Plex is much more geared to streaming, with apps for phones, tablets, game consoles and various IP TV boxes like the Roku. It also has this neat cloud feature where you can watch your media from anywhere with a fast enough internet connection, and share with friends and family. I think it’d be cool to get my parents a Roku box, and be able to ‘curate’ a movie collection for them with no effort on their part and without them having to have a server/NAS running, by ripping their DVDs onto my home NAS.

Another feature of Kodi that has been taken out of Plex is the concept of a ‘stub’ file, where you just keep a 0 byte file on the NAS that says “out in the physical world, there’s a DVD with ‘The Godfather (1972)’ on it”, so that Kodi can be your librarian for offline movies too. So now, to use Plex, I definitely need to have some kind of online copy of each DVD (or maybe a 2 second video to take its place, that says “look on the shelf behind you”… hmmm)

Anyway, I now have a lot of ISO files that need to become MKV or avi or something else, ideally with no input from me. It took me a while to work out how to get a title extracted from a DVD ISO (not a physical DVD) as an MPEG2 stream, so here’s another little post about that. ‘title’ is what the DVD standard calls the individual video streams on the DVD – for example, one for the main movie, one for the trailer, and others for special features. They are all stored in MPEG-2 video format, which is getting old now, and not so great for compression. The cool kids are using H.264 these days.

I did this on my Windows desktop, but the tools are natively Linux tools so it should work on Linux or OS X too. You need mplayer and ffmpeg. Actually, you probably could do this with just mplayer/mencoder, but this worked for me and so I stopped looking. I specifically wanted a command-line method to do it. If you don’t care about that, just use Handbrake, which is very nice. I may yet do that too, since it also has a job queue.

First, to extract Title 1 from the DVD as MPEG-2 video:

mplayer dvdnav://1 -dvd-device DVDVolume.iso -dumpstream -dumpfile title1.m2v

If you don’t know which title you need, mplayer can help there too:

mplayer dvdnav:// -dvd-device DVDVolume.iso -frames 0 -identify -vo null -ao null -nocache

which will spit out a LOT of information, including this:

TITLE 1, CHAPTERS: 00:01:37,00:10:51,00:14:10,00:23:21,00:24:09,00:27:44 {etc}
TITLE 2, CHAPTERS: 00:02:10,

which tells me that title 1 is the movie, and title 2 is the trailer (only 2 minutes long).

Then to make an MKV file of the resulting MPEG-2 file:

ffmpeg.exe -i title1.m2v ./title1.mkv

There are loads of settings for x264 that I don’t understand, so I just ignored them. The results look pretty good so far, and I get files of about 1.5GB instead of the 4GB original. Each one takes a while (about an hour? I haven’t timed it), but there’s nothing to watch or poke, so it doesn’t really matter. I can leave it to do its thing.

Monitoring Network Projects & Hacking

wraprancid and RANCID 3.x

Jethro R Binks’ excellent wraprancid script allows you to bring in configurations (and pretty much anything else that can be text) without having to get involved in writing a new ?rancid/?login combination for your device. That avoids some pretty hairy perl and Tcl code, so it’s definitely a Good Thing! It’s also useful for devices that don’t even have a command-line, but might allow you to fetch their config from a web page, or TFTP.

The trouble is, RANCID changed the way it deals with device types between RANCID 2.x and RANCID 3. It changed in a good way, so that the patches to rancid-fe that tools like wraprancid required are no longer necessary. What was previously hard-coded in the source of rancid-fe is now a proper configuration file, with a second config file for you to add your own types to. Here’s how to get wraprancid working with RANCID 3.x

First, I’m assuming you have a working wrapplugin script. Here’s one I use to fetch the config from Asterisk servers.

#!/opt/perl/bin/perl -w
# Modules

# Load any modules needed
use strict;
use Getopt::Std;
use Net::SSH::Perl;

# Variables

# Initialize variables used in this script

my $debug = 0;

my %options = ();
getopts('df:', \%options);
my $file = $options{'f'};
my $fh;
my $host = $ARGV[0];

$debug = $options{'d'};

print STDERR "to host: $host\n" if $debug;

my $ssh = Net::SSH::Perl->new($host, protocol => '2,1', debug => $debug );

print STDERR "made ssh obj\n" if $debug;

print STDERR "login\n" if $debug;
my ($stdout, $stderr, $exit) = $ssh->cmd("true");
print STDERR "got output\n" if $debug;

# Open the output file.
open($fh, ">", $file) or die "Cannot open output file\n";
print $fh "#RANCID-CONTENT-TYPE: wrapper.asterisk\n#\n";

print $fh $stdout;
print STDERR "wrote output of ". length($stdout)." bytes\n" if $debug;

# End #
print STDERR "done\n" if $debug;

That lives in ~rancid/bin/asterisk.wrapplugin, just as it did in version 2.

Then, in ~rancid/etc/rancid.types.conf, we’ll define a new device type called wrapper-asterisk:

wrapper-asterisk;script;wraprancid -s asterisk.wrapplugin

(I don’t think the login script matters, as it’s never used, but it must be specified to keep RANCID happy)

And finally in the router.db, you can put your actual device:


That’s it. You can repeat for whichever other scripts you need to do this for.

Bonus Tip

The asterisk end of the script above works like this: we use SSH public key authentication to connect to the server, and then in ~root/.ssh/authorized_keys, there is a line like this:

command="/usr/sbin/asterisk -V; echo 'extensions.conf'; cat /etc/asterisk/extensions.conf; echo 'sip.conf'; cat /etc/asterisk/sip.conf; echo 'iax.conf';cat  /etc/asterisk/iax.conf",from="myrancidhost" ssh-dss AAAAB3NzaC174ENozlUVBe5hH32Wy/duAJt1b4nWbVPoW1GP/koSZNv3888s3fx23nEpLMJxispulA== rancid@myrancidhost

So that the user authenticating with that particular key doesn’t get a shell, they just get the output from a series of cat commands, and then disconnected. They must also be connecting from the RANCID server.

So, now we have Asterisk in the same version control system as our network gear. You can use a similar setup for things like BSD ipfw-based firewalls, or Quagga routers.


IOS XR SSH: “Server refused to allocate pty”

Just another SSH note, since it took me a while to work out what was going on: if you putty into an IOS XR router (ASR9001 in my case) with SSH Agent Forwarding enabled then it just drops the connection with this message: “Server refused to allocate pty”

Solution: don’t do that. Turn off Agent Forwarding in the SSH options (or stop using ssh -A for the commandline client).