Terin Stock

Creating Self-Hosted Tile Maps from OpenStreetMap Data

terin@terinstock.com (Terin Stock) — Mon, 19 Jun 2023 21:30:55 +0000

The work-in-progess homepage of NLAlerts, showing the most recent alerts.

I’ve been working on NLAlerts, a project to archive and display the mobile emergency alerts sent by the Netherlands government. The data provided by the government includes the text of the alert, along with targetted geographical area. With JavaScript libraries such as Leaflet or OpenLayers it is easy to turn this geographical area onto an interactive map.

These libraries can be used with external tile servers, such as the one provided by OpenStreetMap or one of the many commercial offerings. However, I wanted self-host all the data required for the website, which includes the map tiles. Rendered tiles can be stored in a SQLite database following the MBTiles Specification. Since the NLAlerts site already uses SQLite by way of being powered by data exploration framework Datasette, using MBTiles would be fairly complimentary. In fact, there’s already a plugin datasette-tiles to have Datasette act as a tileserver.

This just left one remaining problems: How do I get OpenStreetMap data into MBTiles? I found many guides online, but they seemed to fall into one of two camps: they were written many years ago and they suffer from linkrot or bitrot, or they ripped tiles from other servers. The former problem makes it hard to follow these guides in 2023, the latter is often against the terms of service (or rude to the remaining operators of non-profit tileservers).

This past weekend I was able to cobble together a working pipeline to “pre-render” tiles from OpenStreetMap extracts to MBTiles.

I first started by downloading the “osm.pbf” file for the Netherlands from Geofabrik. This file is the OpenStreetMap data in Protobuf format.

I then used the openstreetmap-tile-server container to import the “osm.pbf” file into PostGIS. I created volumes to store the database (osm-data) as well as where the tiles would later be rendered to (osm-tiles).

podman volume create osm-data
podman volume create osm-tiles

podman run -v $PWD/netherlands-latest.osm.pbf \
           -v osm-data:/data/database \
           -v osm-tiles:/data/tiles \
           overv/openstreetmap-tile-server:2.3.0 \
           import

After several minutes, the container should exit successfully and all the data will be imported. The same image could be used to run the renderer. I had to increase the shared memory configured in the container to avoid rendering errors later.

podman run -v osm-data:/data/database \
           -v osm-tiles:/data/tiles \
           -shm-size=1G \
           overv/openstreetmap-tile-server:2.3.0 \
           run

In another terminal we could exec into this container to start the pre-rendering.

podman exec -it --latest bash

Within the container, we can fetch the render_list_geo.pl script, which wraps the render_list command to render within a bounding box at different zoom levels. I then ran the script with the bounding box determined with the Bounding Box Tool for zoom levels 8 through 13.

wget https://raw.githubusercontent.com/alx77/render_list_geo.pl/master/render_list_geo.pl
perl ./render_list_geo.pl -z 8 -Z 13 \
    -x 3.3316001 -X 7.2275102 \
    -y 50.7503838 -Y 53.6316

If all went well, we can exit and stop the container, now having completed the hardest part. We have the map rendered in the osm-tiles, unfortunately not in a format we can directly use, but instead in meta tile, which can be efficiently used by mod_tile.

Fortunately, Geofabrik has a tool to help us out, meta2tile, which can be used to generate MBTiles databases. After cloning the repository, I built the tool in the most barebones configuration, only enabling MBTiles support.

gcc -DWITH_MBTILES meta2tile.c -o meta2tile -lsqlite3 -lcrypto -lm -O3

It’s worth noting that meta2tile uses the deprecated MD5 function from OpenSSL, which may stop being provided in a future version of OpenSSL. Fortunately, it’s also not the most difficult function to replace.

On my system Podman runs in rootless mode, with volumes being plain directories on disks. Thus I was able to run meta2tile directly against this directory. You may need to mount the volume or copy the data out first.

meta2tile --mbtiles \
          --meta name="Netherlands" \
          --meta type=raster \
          --meta format=png \
          --meta version=1.0 \
          --meta bounds="3.3316001,50.7503838,7.2275102,53.6316" \
          --meta description="OpenStreetMap tiles for Netherlands"
          /home/terin/.local/share/containers/storage/volumes/osm-tiles/_data/default/ \
          nl.db

I was then able to start Datasette with the datasette-tiles plugin installed and browse to http://localhost:8001/-/tiles/nl/ and browse my tiles.

The alert page showing a test alert sent, rendering the targetted area with OpenLayers.

When creating my template for the custom alert pages, I was able to reference "/-/tiles/nl/{z}/{x}/{y}.png" as the location of the tileserver. The result can be seen on this test alert.

Controlling the XM PCR receiver from Linux

terin@terinstock.com (Terin Stock) — Mon, 08 May 2023 20:04:12 +0000

The XM PCR in question, complete with mysterious screw holes.

The XM PCR was a satellite radio receiver sold to consumers in 2003 that was controllable by a PC running Windows over USB. Other than satellite, audio and USB connectivity, the receiver does not have any form of user interactivity, everything is done over USB. XM Radio discontinued it in 2004 due to software to easily rip songs, which was discussed on The Tech Guy #71.

It’s been nearly 20 years since the device was released, and the SiriusXM satellites are still broadcasting audio in a format this receiver can understand. I was shocked when the activation signal went through and the radio sprang to life. With the radio working, audio compression artifacts and all, I sought to control it from Linux.

The receiver uses an FTDI component to expose a serial port over USB. This is supported by modern Linux, and after attaching the receiver to the computer, a serial device was exposed under /dev. This was off to a promising start.

After a little bit of investigation I found that the protocol had already been reverse engineered and Michael Minn had released a GUI program MMXMPCR in 2003. It was last updated in 2005 which was a long time ago in the realm of Linux desktop software. I wasn’t even sure if modern distributions still shipped the required libraries, and if they did, if the program would even compile. Fortunately, Michael still hosts the source tarballs on their website; I downloaded the latest release to find out.

For the moment, Linux distributions still ship the necessary MOTIF and X Toolkit Intrinsics libraries. As desktop Linux continues the migration to Wayland, I wonder how much longer this will hold true. I modified the makefile to specify the correct header location and the linker flag for the Intrinsics library. I ran make and was greeted with a mmxmpcr binary, and a wall of compiler warnings.

Screenshot of mmxmpcr, showing the first 10 channels in the application window.

Upon running mmxmpcr, a window appeared which began populating with channel information from the radio, and I could control the receiver without issue. To remain cool, I tuned into TikTok Radio and attempted to dance.

In order to preserve the project into the future, I’ve used the available release tarballs to create the mmxmpcr repository on GitHub. It would be interesting to refactor the project to use libusb to discover an attached radio, rather than hardcoding a path into the binary.

However, the original Windows software distributed by XM Radio for this receiver seems missing from the Internet Archive. If you have it, please upload it and let me know!

HPSS-Disassembly Progress Report (May 2022)

terin@terinstock.com (Terin Stock) — Tue, 31 May 2022 16:16:16 +0000

Introduction

Title screen of Harry Potter and the Sorcerer’s Stone for the Game Boy Color

A little over 20 years ago the first movie in the Harry Potter series, Harry Potter and the Sorcerer’s Stone in the United States, was released by Warner Bros. I had been caught up in the Harry Potter mania and received the movie tie-in Game Boy Color game as a Christmas gift. I recall playing it for a while, before my attention returned to the other hype machine of 2001: the Pokémon series.

Harry Potter and the Sorcerer’s Stone on PC. Image credit Federico Dossena.

The PlayStation and PC versions of the tie-in game are 3D action-puzzle games where you play as the titular Harry Potter as he navigates around Hogwarts, the grounds, the Forbidden Forest, and a side quest to Diagon Alley. The PlayStation version is where we get the low-poly meme Hagrid from.

Harry Potter fighting bats in a Final Fantasy-inspired battle system.

The Game Boy Color version was completely different: a top-down Role Playing Game in the style of classic Final Fantasy. While you still play as Harry Potter, it follows the plot of the book much more faithfully than the 3D counterparts. There are some changes to adapt into a game: spells are learned for combat, items can be equipped for buffs, and wizard cards can be collected and used for combat effects.

I had entirely forgotten about this game until YouTube channel Flandrew put out a comparison of every version of Harry Potter and the Philosopher’s Stone¹, jogging back a flood of forgotten memories.

Picking it up again, I was excited by how colorful, expansive, and smooth the game felt, especially compared to Pokémon titles of the time. I decided I would learn more about how the game ticked.

Disassembly Beginnings

Disassembling Game Boy games isn’t new; Pokémon Red and Blue has been completely reverse engineered (this has prompted projects disassembling the rest of the Pokémon series). Other classics, such as Zelda: Link’s Awakening DX, have also been in progress for years.

I’ve started a project to disassemble Harry Potter and the Sorcerer’s Stone for the Game Boy Color. My goal is to document the techniques used to develop Game Boy games during the final years of the hardware lifecycle. I’m also planning on taking techniques learned from this project forward towards disassembling the sequel, Harry Potter and the Chamber of Secrets, which also holds the distinction of being the last Game Boy Color game released in North America.²

As I’ve never worked on disassembling a game before, I had to start from scratch. Fortunately, the Game Boy homebrew development scene is quite active, so modern tools are available to help get started.

mgbdis, the disassembler

To begin understanding what’s happening “behind the curtain” we need to turn the binary code the Game Boy Color CPU executes back into assembly code.

Unfortunately, the labels and symbols are not preserved in the binary code, so we’ll never be able to reproduce the exact structure the original game developers had for the game. However, we can use computers to make best guesses as a starting point, and use logic and reason (and our own guesses) to craft a structure back on top.

I used Matt Currie’s mgbdis disassembler to begin this task. Sorcerer’s Stone is a 4 MiB game, so disassembling took a while, but it ended up creating 256 files each representing 16 KiB of bankable memory. I also attempted to use Matt’s emulator, Beaten Dying Moon, to automate generating a symbol file to aid in separating executing code from data like images, but it did not seem very successful.

RGBDS, the assembler and linker

Once we have disassembled the game (even with our imperfect results) in order to get back to a playable game again we’ll need an assembler and linker. Fortunately we have RGBDS, an open source toolchain dating back to the 1990s.

RGBDS’s assembler, rgbasm, takes the assembly files as inputs and turned them into object files. It’s likely down the road we’ll have object files containing code that are logically related to each other, but for now we assemble each bank into it’s own object file.

RGBDS’s linker, rgblink, collects the object files together and decides how to combine them together into a Game Boy ROM. As the code is “fixed” in memory, the primary responsibility of the linker in this project is to resolve symbol references across object files, so the correct memory locations can be written into the ROM. As we cleanup the assembly files, the linker will likely become more important.

RGBDS includes a header fixer, rgbfix, used to generate a valid Game Boy header. The original hardware uses information in this header as checksums, to setup compatibility modes, and to implement basic DRM with the “Nintendo” logo. When the game was disassembled this header was partially decoded as instructions, and partially as data. It was removed and replaced by padding so rgbfix could be used instead. This allows for greater flexability in generating debug builds later, as the tool can generate the correct checksums later.

One other included tool is RGBDS’s image converter, rgbgfx. This is a tool for storing graphics as PNGs instead of 2bpp, an encoding more suitable for the Game Boy’s hardware. mgbdis did not separate the game’s images into individual files, so we won’t be using it for now, but it will be indespensible later once we’ve extracted the images.

The RGBDS project also documents the object file format, allowing for project specific tools to be written (if that proves to be necessary). Since Sorcerer’s Stone is a JRPG, there is a lot of dialog, menus, and world building, resulting in lots of text. The game also supports 11 different languages, farther multiplying the amount of text. I suspect we might need tooling to easily handle all of it.

gup, the recursive build system

After disassembling a game, mgbdis generates a basic GNU Make-compatible Makefile. This Makefile calls rgbasm over a single assembly file “game.asm” that simply includes all the banks. For a small game this might work, but for a large game like Sorcerer’s Stone re-assembling the entire game each rebuild was actually taking a significant amount of time. It would be far better to only reassemble the files that changed, and the targets that depend on that file.

I’ve switched the project to using gup, a recursive build system inspired by Daniel J. Bernstein’s redo. In gup, targets are executable scripts written in any language, and they can discover and register their own dependencies.

In HPSS-Disassembly, gup assembles banks by calling a script that calls rgbasm and registers each file it lists as a dependency with gup. When a file is changed, gup knows it only needs to rebuild banks that included it the previous build. This makes the testing iteration cycle extremely fast.

Disassembly Progress

I’ve written a lot of words here about the progress made on project infrastructure. What have I actually accomplished on the disassembly side? Since this is my first month working on the project, I’m afraid I haven’t accomplished too much.

Ready? Let’s Start

Error displayed when the game is inserted into a system not compatible with the Game Boy Color.

As Harry Potter and the Sorcerer’s Stone is a Game Boy Color-only game, the very first thing it does is check to see if it’s running on a Game Boy Color-compatible system. When it jumps to Start it compares the value left in the a register by the system boot ROM to $11. If the zero flag is set, it later jumps to code to show an error message.

Start::
    and a                       ; clear flags
    cp BOOTUP_A_CGB             ; is Game Boy Color?

    ld a, $00                   ; set a to 0
    jr nz, .notGBC              ; if not GBC:
    inc a                       ;   increment a (a=1)

.notGBC:
    ldh [$ef], a                ; save GBC value
    ld sp, $cfff                ; setup stack pointer
    ldh a, [$ef]
    or a
    call z, Unknown_Non_GBC     ; call if not GBC

As this code saves the GBC status into High RAM, instead of directly jumping to code to display the error message, I wonder if at some point in development the game was targetting compatibility with the earlier Game Boy systems.

–floop-flatten

What do you do if you want quickly copy contiguous memory from multiple parts of the game, but part of the game needs to copy a different number of bytes, and you also want to keep the number of CPU cycles to a minimum? One approach taken by the Sorcerer’s Stone’s developers is to flatten the loop. You can call the same 3 instructions 32 times, then call into the code at whatever point has the required number of iterations remaining.

Fortunately, rgbasm is a macro assembler, allowing us to refactor this logic to generate the assembly for us, while also generating more readable names.

FOR V, 32, 0, -1                ; loop from 32 to 0, decrementing each time
CopyHL2DE_{d:V}:                ; generate a label we can reference from other code
    ld a, [hl+]                 ; load the byte pointed to by hl into a,
                                ;     and also increment hl
    ld [de], a                  ; load a into the byte referenced by de
    inc de                      ; increment de
ENDR

This reduced over 200 lines of code into a much more manageable 6!

What’s Next

I’ve still have a lot of tasks ahead. Some immediate tasks to start working on:

Begin extracting tiles into bitmaps, and convert and assemble on-the-fly during builds.
Work on extracting text into forms that can be easier to work with, especially for translators.
Label and comment even more code.

In the first blog post for disassembling Zelda: Link’s Awakening DX, Pierre writes:

Reverse-engineering assembly code is quite slow, but I’ll try to post some findings on this blog.

The statement is just as true here as it was all those years ago. We’ll see how this goes!

Flandrew must be from outside the United States of America. ↩︎
According to Wikipedia five games released after Chamber of Secrets in other markets, 1 in Germany, 1 in Korea, and 3 in Japan, including the last licensed game Doraemon no Study Boy: Kanji Yomikaki Master. ↩︎

Systemd's clock-epoch for RTC-less systems

terin@terinstock.com (Terin Stock) — Sun, 26 Dec 2021 06:24:00 +0000

Earlier today I restarted a Raspberry Pi I use on my home network. When the system came up after reboot, many services failed to resume, which I quickly noticed due to the lack of DNS resolution on my network. Upon checking the system logs, the culprit became clear: CoreDNS was failing to verify the certificates of my DNS-over-TLS provider. According to my Raspberry Pi, we were partying like it was 1999. Without DNS resolution the system time would never automatically synchronize with NTP.

On a Raspberry Pi running Raspian or Raspberry Pi OS, the system is automatically configured to use fake-hwclock to save the time to disk from a cron, and restore it at boot. However, this package isn’t widely distributed outside Debian and it’s derivatives.

At boot, systemd compares the system time to a builtin epoch, usually the release or build date of systemd. If it finds the system time is before this epoch, it resets the clock to the epoch. Unfortunately, this is still sometimes too old, especially when it comes to TLS certificates.

We can still utilize this systemd feature however, at least if systemd is version 247 or newer, by utilizing a hook added to aid system administrators and image distributors. If the file /usr/lib/clock-epoch exists, systemd uses the modified time as the systemd epoch.

We can even script periodically updating this file. First, create /etc/systemd/system/set-clock-epoch.service:

[Unit]
Description=Updates the mtime of clock-epoch

[Service]
ExecStart=/bin/touch -m /usr/lib/clock-epoch

Then, create a timer to start it periodically, at /etc/systemd/system/set-clock-epoch.timer:

[Unit]
Description=Timer for updating clock-epoch

[Timer]
OnBootSec=5min
OnUnitInactiveSec=17min

[Install]
WantedBy=timers.target

I’ve arbitrarily delayed activation of this timer by setting OnBootSec, since this is low priority at boot time.

Then reload the daemon, then enable and start the timer.

systemctl daemon-reload
systemctl enable --now set-clock-epoch.timer

Now when the system reboots, systemd will read the modification time of this file during early boot, setting the time to a reasonably close value, allowing certificates to be validated and NTP to take over.

As gioele on Lobste.rs points out, on systemd version 250 and later, if you’re using systemd-timesyncd you can set SaveIntervalSec= to automatically write the time out to the filesystem. This follows a similar mechanism, the file /var/lib/systemd/timesync/clock has it’s modification time set, and is used to restore out reboot. The time is set by the epoch mechanism much earlier in the startup process than the timesyncd mechanism (which first requires service activation to begin). It also doesn’t help if, like me, you’re not using timesyncd.

Clearing Custom Search Engines in Chrome

terin@terinstock.com (Terin Stock) — Wed, 18 Aug 2021 09:20:00 +0000

In addition to your search engine in Google Chrome, the browser tries to be helpful by remembering custom search engines for websites you visit in the course of navigating the web. As some websites can become registered simply by navigating to their homepage, this can result in a very long list of custom search engines.

Unfortunately, the page to manage custom search engines in Google Chrome (currently located at chrome://settings/searchEngines) leaves a lot to be desired. This page has no mechanism for bulk editing. With over two hundred engines registered, I did not want to click my mouse to manually manage them.

Fortunately, we can activate super developer powers!

const nl = document.querySelector("settings-ui").shadowRoot
  .querySelector("#main").shadowRoot
  .querySelector("[role=main]").shadowRoot
  .querySelector("settings-search-page").shadowRoot
  .querySelector("settings-search-engines-page").shadowRoot
  .querySelector("#otherEngines").shadowRoot
  .querySelectorAll("settings-search-engine-entry");

Ah, right. So, the settings pages uses Shadow DOM, which makes getting to the actual list of custom search engines a bit of a chore. Query through a bunch of document fragments to get to the actual list.

const ops = Array.from(nl).map((elem) => elem.shadowRoot)
  .filter((elem) => {
    return !elem.querySelector("#keyword-column").innerText.startsWith("!");
  })
  .map((elem) => {
    return [elem.querySelector("#keyword-column").innerText, elem.querySelector("#delete")];
  });

console.table(ops);

I use the custom search engine feature to implement DuckDuckGo-style !Bang searches. So before removing engines, I filter out items starting with “!”.

If the table printed to the console looks good, click the virtual mouse.

ops.forEach(entry => entry[1].click());

Setting up a git server on NixOS

terin@terinstock.com (Terin Stock) — Wed, 27 Jan 2021 21:23:25 +0000

Over the years I’ve tried out many VPS providers, which has resulted in personal infrastructure spread far and wide. I’ve decided to start 2021 by consolidating this infrastructure, where reasonable. This also gives me an opportunity to properly document deployments, ideally making them easy to redeploy in the future. NixOS allows me to solve both goals relatively easily.

One piece of personal infrastructure is a git server, git.terinstock.com, which I use for personal configuration or smaller projects. Since I’m the only user, I don’t need the fancy interfaces provided by options like gogs, gitea, or GitLab. I’ve chosen to use cgit, which had a few more features I wanted over gitweb.

It did feel a little strange to deploy a cgi server in 2021. I think we call this “serverless” now?

cgit

NixOS already has cgit packaged, so we can start with configuring it.

let
  cgit = pkgs.cgit;
  cgitConfig = pkgs.writeText "cgitrc" (lib.generators.toKeyValue { } {
    css = "/cgit.css";
    logo = "/cgit.png";
    favicon = "/favicon.ico";
    about-filter = "${cgit}/lib/cgit/filters/about-formatting.sh";
    source-filter = "${cgit}/lib/cgit/filters/syntax-highlighting.py";
    clone-url = (lib.concatStringsSep " " [
      "https://$HTTP_HOST$SCRIPT_NAME/$CGIT_REPO_URL"
      "ssh://git@git.terinstock.com:$CGIT_REPO_URL"
    ]);
    enable-log-filecount = 1;
    enable-log-linecount = 1;
    enable-git-config = 1;
    root-title = "git.terinstock.com";
    root-desc = "Terin's Git Repositories";
    scan-path = "/srv/git";
  });

This assigns a derivation to cgitConfig that when evaluated would create the ini-like configuration file. Most of this comes down to personal preference with a few exceptions:

about-filter and source-filter reference the respective filters wpithin the cgit package. NixOS will expand these to full paths when creating the configuration file.
scan-path is the location on disk I’m using to host the git repositories.

git-shell

To allow for authenticated pushes, I use git-shell provided by the git project. This provides a minimal “shell” that an execute a prescribed list of git commands. I define a system user git and assign it this shell.

{
  users.users.git = {
    isSystemUser = true;
    description = "git user";
    home = "/srv/git";
    shell = "${pkgs.git}/bin/git-shell";
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIINNSIl3/j3KqW/x3kFj1ZvZlSxp+MDhk8LAIDlqs/9w"
    ];
  };
}

As I’m the only user that will be pushing to these repositories, I don’t need to configure anything extra for authenticating actions.

Web Server, h2o

Since cgit is a cgi application, it needs a web server to actually handle the HTTP connections. I’ve chosen to use h2o as it supports being an application proxy in addition to serving static files, while still being configurable with YAML/JSON.

let h2oFile = file: {
    "file.file" = "${file}";
    "file.send-compressed" = "ON";
  };
  h2oHeaderList = attrs: (lib.mapAttrsToList (k: v: "${k}: ${v}") attrs);
  h2oConfig = pkgs.writeText "h2o.conf" (lib.generators.toYAML { } {
    hosts."git.terinstock.com" = {
      paths = {
        "/cgit.css" = h2oFile ./cgit.css;
        "/cgit.png" = h2oFile "${cgit}/cgit/cgit.png";
        "/favicon.ico" = h2oFile "${cgit}/cgit/favicon.ico";
        "/robots.txt" = h2oFile "${cgit}/cgit/robots.txt";
        "/" = {
          "fastcgi.spawn" = "${pkgs.h2o}/share/h2o/fastcgi-cgi";
          setenv = {
            SCRIPT_FILENAME = "${cgit}/cgit/cgit.cgi";
            CGIT_CONFIG = "${cgitConfig}";
          };
          compress = "ON";
        };
      };
      "header.set" = {
        header = (h2oHeaderList {
          x-frame-options = "deny";
          x-xss-protection = "1, mode=block";
          x-content-type-options = "nosniff";
          referrer-policy = "no-referrer, strict-origin-when-cross-origin";
          cache-control = "no-transform";
          strict-transport-security = "max-age=63072000";
          content-security-policy = (lib.concatStringsSep "; " [
            "default-src 'none'"
            "style-src 'self' 'unsafe-inline'"
            "img-src 'self' data: https://img.shields.io"
            "script-src-attr 'unsafe-inline'"
          ]);
          expect-ct = "enforce, max-age=30";
        });
      };
      listen = {
        port = 443;
        ssl = {
          certificate-file = "/var/lib/acme/git.terinstock.com/fullchain.pem";
          key-file = "/var/lib/acme/git.terinstock.com/key.pem";
          min-version = "TLSv1.2";
          cipher-preference = "server";
          cipher-suite = (lib.concatStringsSep ":" [
            "TLS_AES_128_GCM_SHA256"
            "TLS_AES_256_GCM_SHA384"
            "TLS_CHACHA20_POLY1305_SHA256"
            "ECDHE-ECDSA-AES128-GCM-SHA256"
            "ECDHE-RSA-AES128-GCM-SHA256"
            "ECDHE-ECDSA-AES256-GCM-SHA384"
            "ECDHE-RSA-AES256-GCM-SHA384"
            "ECDHE-ECDSA-CHACHA20-POLY1305"
            "ECDHE-RSA-CHACHA20-POLY1305"
            "DHE-RSA-AES128-GCM-SHA256"
            "DHE-RSA-AES256-GCM-SHA384"
          ]);
        };
      };
    };
  });

Like with cgit’s configuration, h2oConfig is assigned a derivation that when evaluated will create a configuration file for h2o. In this configuration one host is defined and paths are defined for static assets, A fastcgi handler is configured for the root, which will be used for all paths not matched by a static file path.

h2o does not support cgi directly, but ships with a script to proxy through a fastcgi server. This wrapper is configured with the path to cgit’s and the cgit configuration file defined earlier.

This is more verbose than strictly neccessary, as I wanted to have high marks on Qualys’s SSL Labs report and Security Headers. I imagine in the future I’ll have refactored this into smaller functions and options.

The documented syntax for adding response headers involves setting the headers.set key multiple times. This is not representable in Nix, as Nix will not allow the same key to be set multiple times. An undocumented syntax using a YAML sequence instead has been available since 2.3.0-rc2.

header.set:
    header:
        - "x-frame-options: \"deny\""
        - "x-xss-protection: \"1, mode=block\""

Finally, I we need to start the h2o web server. At the time of writing, h2o does not have a NixOS module, but we can use lower-level modules ourselves.

{
  systemd.services.h2o = {
    description = "H2O web server";
    after = [ "network-online.target" "acme-git.terinstock.com.service" ];
    wantedBy = [ "multi-user.target" ];
    path = [ pkgs.perl pkgs.openssl ];
    serviceConfig = {
      ExecStart = "${pkgs.h2o}/bin/h2o --mode master --conf ${h2oConfig}";
      ExecReload = "${pkgs.coreutils}/bin/kill -s HUP $MAINPID";
      ExecStop = "${pkgs.coreutils}/bin/kill -s QUIT $MAINPID";
      User = "h2o";
      Group = "h2o";
      Type = "simple";
      Restart = "on-failure";
      AmbientCapabilities = "cap_net_bind_service";
      CapabilitiesBoundingSet = "cap_net_bind_service";
      NoNewPrivileges = true;
      LimitNPROC = 64;
      LimitNOFILE = 1048576;
      PrivateDevices = true;
      PrivetTmp = true;
      ProtectHome = true;
      ProtectSystem = "full";
      ReadOnlyPaths = "/var/lib/acme/";
      ReadWriteDirectories = "/var/lib/h2o";
    };
  };
}

This configures a systemd unit that will be enabled at startup. The ExecStart will start h2o without daemonizing, and provides the path to the evaluated h2o configuration. The PATH environment variable for the service will be updated to include Perl (for the fastcgi proxy) and OpenSSL (for OCSP stapling). I may submit a pull request to nixpkgs to fix the h2o derivation to reference these directly.

Additional settings are provided to the unit to configure systemd’s sandbox for the service.

{
  users.users.h2o = {
    group = "h2o";
    home = "/var/lib/h2o";
    createHome = true;
    isSystemUser = true;
  };

  users.groups.h2o = { };
}

Another system user is created, this time for h2o, along with a correspondng group used shortly.

ACME Certificates

NixOS ships with great support for using ACME to create TLS certificates, such as from Let’s Encrypt or your own certificate authority.

{
  security.acme = {
    # Set to true if you agree to your ACME server's Terms of Service.
    # For Let's Encrypt: https://letsencrypt.org/repository/
    acceptTerms = false; 
    email = "terinjokes@gmail.com";
    certs = {
      "git.terinstock.com" = {
        dnsProvider = "cloudflare";
        credentialsFile = "/var/lib/secrets/cloudflare.secret";
        group = "h2o";
      };
    };
  };
}

To avoiding needing to configure h2o to proxy to lego for HTTP challenges, I prefer to use the DNS ACME challenges with lego’s support for Cloudflare DNS. Configuring lego to do DNS challenges is outside the scope of this post.

Housekeeping

I configure a few more NixOS options for general housekeeping that don’t fit in the above sections.

{
  environment.systemPackages = [ pkgs.git ];
}

I install git as a system package so it’s avialable should I need to log into the box to handle something.

{
  services.openssh.passwordAuthentication = false;
  services.sshguard.enable = true;
}

I disable OpenSSH’s password authentication mechanisms, as I have a strong preference to using more secure options. I also enable sshguard to block connections that try to log in with a password anyways.

{
  nix.gc = {
    automatic = true;
    options = "--delete-older-than 30d";
  };
  nix.optimise.automatic = true;
  system.autoUpgrade = {
    enable = true;
    allowReboot = true;
  };
}

Configures NixOS to perform routine maintaince in the background. This allows for NixOS to update at a regular interval, rebooting if needed to install kernel updates, as well as optimizing and garbage collecting the nix store.

While the goal in 2021 is to have less machines to manage, this ensures I don’t forget to install security patches because I haven’t logged in for a while.

Pebble and lego to test ACME with NixOS

terin@terinstock.com (Terin Stock) — Thu, 14 Jan 2021 22:04:00 +0000

When configuring a new service I want to run on NixOS, I often use nixos-shell to quickly standup a temporary VM locally with my new configuration. When this configuration is a service, I often want to ensure I have TLS configured properly, including the correct permissions on the certificate and key files managed by lego.

However, I don’t neccessarily want to send unacceptable amounts of traffic to production ACME servers, or deal with proper validation with their staging services. Fortunately, we can configure the NixOS VM to start a testing ACME server, Pebble, and configure Lego to use it.

Pebble

Pebble is a small, single-binary ACME server intended for testing. Keys and certificates are randomnized between calls, but this is fine for an emphermial VM.

First, we’ll want to configure Pebble to start, which we can do with the systemd.service NixOS option. I use the toJSON builtin function to create a JSON configuration file for Pebble from a Nix attribute set. I also reference the default certificate and key from the source package, as they are not yet copied to the output package.

{ pkgs }:

let
  pebbleConfig = pkgs.writeText "pebble.json" (builtins.toJSON {
    pebble = {
      listenAddress = "0.0.0.0:14000";
      managementListenAddress = "0.0.0.0:15000";
      certificate = "${pkgs.pebble.src}/test/certs/localhost/cert.pem";
      privateKey = "${pkgs.pebble.src}/test/certs/localhost/key.pem";
      httpPort = 5002;
      tlsPort = 5001;
      ocspResponderURL = "";
      externalAccountBindingRequired = false;
    };
  });
in {
  systemd.services.pebble = {
    description = "Pebble ACME Test Server";
    after = [ "network-online.target" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      Environment = [ "PEBBLE_VA_NOSLEEP=1" "PEBBLE_VA_ALWAYS_VALID=1" ];
      ExecStart = "${pkgs.pebble}/bin/pebble -config ${pebbleConfig}";
      DynamicUser = true;
    };
  };
}

You can modify some behavior of Pebble through environment variables. I set two for better behavior in a development VM:

PEBBLE_VA_NOSLEEP disables any artifical sleeps in the issuance path, as we’re not interested in testing lego’s validationg polling.
PEBBLE_VA_ALWAYS_VALID disables all validation methods, and assumes domains have already been successfully validated.

One warning from the Pebble documentation bares repeating here:

Pebble is NOT INTENDED FOR PRODUCTION USE. Pebble is for testing only.

lego

lego is a widely used ACME client that implements all of the ACME challenges, bindings for major DNS providers, and support for bundling certificates. It’s used as the implementation to NixOS’s security.acme options. This means most of the configuration is already done for us.

{
  security.acme = {
    server = "https://localhost:14000/dir";
    acceptTerms = true;
    email = "webmaster@example.com";
    certs = {
      "example.com" = {
        webroot = "/var/www/example.com";
        group = "prosody";
      };
    };
  };
}

Despite not needing to implement any challenges, as we’ve disabled them in Pebble, we still need to provide webroot configuration. The group attribute is used to set the group permission on the generated certificates and keys, and should be set to the same user your server is running as.

There’s one farther complication: lego does not trust the certificate authority used by the ACME server, and thus it won’t issue requests out of the box. We can configure the lego to trust these certificates by setting the environment of the generated service unit.

{
  systemd.services."acme-example.com".serviceConfig.Environment = [
    "LEGO_CA_CERTIFICATES=${pkgs.pebble.src}/test/certs/pebble.minica.pem"
    "LEGO_CA_SERVER_NAME=localhost"
  ];
}

Usage

The certificates and keys created by security.acme are stored underneath /var/lib/acme/. This can be provided to your server’s configuration.

{
  services.prosody = {
    enable = true;
    virtualHosts."example.com" = {
      enabled = true;
      ssl.cert = "/var/lib/acme/git.terinstock.com/fullchain.pem";
      ssl.key = "/var/lib/acme/git.terinstock.com/key.pem";
    };
  };
}

K9P: Kubernetes as 9P Files

terin@terinstock.com (Terin Stock) — Wed, 20 Nov 2019 11:55:00 -0800

K9P is a virtual filesystem that allows you to interact with Kubernetes clusters via a 9P filesystem, allowing us to continue to believe that everything is a file. Presented at KubeCon + CloudNativeCon North America 2019.

Building a Hardware MIDI Player

terin@terinstock.com (Terin Stock) — Sat, 02 Feb 2019 15:30:00 +0100

Building a MIDI player using Go and various Linux system interfaces, presented at FOSDEM 2019.

memfd_create: Temporary in-memory files with Go and Linux

terin@terinstock.com (Terin Stock) — Sat, 20 Oct 2018 23:21:27 +0000

My MIDI Player project requires interfacing with libraries and system calls that don’t yet have Go equivalents. A few of these C libraries expect to be invoked with file paths or file descriptors.

If the data is already available as a file on a mounted disk, this is fine, I can pass the path or descriptor that already exists. I run into a problem, however, as soon as I want to work with data that isn’t directly backed by a file, such as after being manipulated by the program, or having been unpacked from a container format.

There’s a simple way to get a file path: create a temporary file somewhere. This requires having a writable filesystem mounted, and cleaning up the files when no longer required. On a device that’s intended to be used as an appliance, I don’t have anywhere writable.

In Linux 3.17 and later, I have another option memfd_create(2):

memfd_create() creates an anonymous file and returns a file descriptor that refers to it. The file behaves like a regular file, and so can be modified, truncated, memory-mapped, and so on. However, unlike a regular file, it lives in RAM and has a volatile backing storage. Once all references to the file are dropped, it is automatically released.

This checks the boxes for most of my requirements: a file descriptor to something that acts like a normal file, backed by memory, doesn’t require any writable mount points, and automatically cleans up files when closed.

The system call is available in the Go x/sys/unix package, along with a few companion calls.

package main

import (
	"fmt"

	"golang.org/x/sys/unix"
)

// memfile takes a file name used, and the byte slice
// containing data the file should contain.
//
// name does not need to be unique, as it's used only
// for debugging purposes.
//
// It is up to the caller to close the returned descriptor.
func memfile(name string, b []byte) (int, error) {
	fd, err := unix.MemfdCreate(name, 0)
	if err != nil {
		return 0, fmt.Errorf("MemfdCreate: %v", err)
	}

	err = unix.Ftruncate(fd, int64(len(b)))
	if err != nil {
		return 0, fmt.Errorf("Ftruncate: %v", err)
	}

	data, err := unix.Mmap(fd, 0, len(b), unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)
	if err != nil {
		return 0, fmt.Errorf("Mmap: %v", err)
	}

	copy(data, b)

	err = unix.Munmap(data)
	if err != nil {
		return 0, fmt.Errorf("Munmap: %v", err)
	}

	return fd, nil
}

Some libraries require a file path instead of a file descriptor. Fortunately, we can turn a file descriptor into a path by way of the proc filesystem using the symlinks in the /proc/self/fd directory.

package main

import (
	"fmt"
	"log"
	"os"
)

func main() {
	fd, err != memfile("hello", []byte("hello world!"))
	if err != nil {
		log.Fatalf("memfile: %v", err)
	}

	// filepath to our newly created in-memory file descriptor
	fp := fmt.Sprintf("/proc/self/fd/%d", fd)

	// create an *os.File, should you need it
	// alternatively, pass fd or fp as input to a library.
	f := os.NewFile(uintptr(fd), fp)
	defer f.Close()
}

Eventually, I’ll replace these C libraries with Go packages that operate on io.Reader, and this workaround will become unnecessary. Until then, I’m glad the option is available.

Hardware MIDI Player (Part One)

terin@terinstock.com (Terin Stock) — Mon, 17 Sep 2018 00:35:55 +0000

Overview

Raspberry Pi MIDI Player

Last December, I watched Techmoan’s YouTube video on the Roland MT-80S, a player built to help teach people how to play songs, that used HD 3.5" floppy disks as storage medium for MIDI files. Since then AkBKukU has done a video on the Yamaha Disk Orchestra DOU-10, that used DD disks instead.

As I enjoy listening to chiptune and DOS-era video game music, I was keen on having a player in my collection. Cursory searches for players sold online resulted in typically high eBay prices. Considering my growth in ability with electronics in the past year, I decided to forgo the market and build one myself.

The vision in my mind belongs in the mid-90s. My completed player shouldn’t feel out of place alongside a tape deck and CD changer. The insides, however, should be built from modern components.¹

Hardware

I began with the internal hardware, which is somewhere underneath all those wires.

I’ve built the project using Raspberry Pi 3B. This is significantly more powerful than required for playing back MIDI files.

In fact, I had originally planned to use an Arduino Uno as a rudimentary central processor and floppy drive controller. However, I was unable to reliably control any of the floppy drives shown here—they might not even work properly—and decided to switch to using a USB floppy drive that I could more easily verify. I switched to the Raspberry Pi because it is a better USB host than an Arduino.

Maccally USB Floppy Disk Drive for Mac

I settled on a rather generic USB floppy drive. Well, not just any USB floppy drive, but one “for Mac”. Based on the stripe design, I’m guessing it was meant to pair (or, pear) with the fruit-colored iBook G3 clamshells. Fortunately, the plastic case comes off simply enough.

This choice of drive proved to be troublesome as I would soon find myself on a pretty significant detour as a result.

I used the Protoboard hat from MakerSpot for the remainder of my hardware work. The Pi’s serial data and power lines are connected through to a 90s-era WaveBlaster-compatible 2x13² header. I’m used an absolutely tiny Dreamblaster S2 as a modern General MIDI synth. The output from the synthesis chip, a SAM2695, was routed to the Pi’s 3.5mm AV jack. A proper quarter-inch jack has been ordered.

Software

As a software guy, I expected the hardware to be the hard part. As Caesar would have said, “The fault, dear Brutus, is not in our stars, but in our software.”³

So far I’ve only written a rather trivial MIDI player. The path to an SMF or RIFF encoded MIDI file is provided as the command line argument, the software decodes and at the appropriate time sends the MIDI event to the S2 daughterboard for synthesis. I also confirmed manual playback of files on a floppy disk worked.

Shortly thereafter, I switched from Raspbian to a stripped down Linux distribution. Much to my horror, I found the floppy drive no longer worked. Any attempt to mount or otherwise interact with the floppy drive would completely freeze the running program, and no amount of SIGINT or SIGTERM would regain control. The floppy drive had to be disconnected from the Pi.

I spent the next several weeks debugging why the floppy drive stopped working. I started with debugging the floppy drive itself, then confirmed the power supply was outputting the correct voltages and was clean. I even recompiled the Linux kernel with different options!

Eventually, I discovered the drive worked with the 4.14 Linux kernel, but stopped working some point thereafter. Facing an 8-month range of changes to the kernel, I turned to git-bisect and started a week-long hunt for the breaking change. Since this range was so large, and the underlying files so different, setting up a compilation cache did not decrease the rebuild durations.

After at least 16 bisection points, I was lead to a single commit in the kernel driver for the USB controller used by the Raspberry Pi, the DesignWare USB2. In this change, a communication delay was added after multiple successive NAKs in the USB protocol, to allow low-speed Chromebooks (and other ARM-based devices) to do other work, instead of being tied up in an interrupt loop.

The commit requests a 1 millisecond delay, but uses the kernel’s low-resolution timer API. On devices where this timer is coarse, like the Raspberry Pi, this delay might actually be closer to 10-20 milliseconds. Fortunately, most USB devices tolerate delays in the communication. Unfortunately, this drive is not one that does so. This drive only tolerates delays of up to 5 milliseconds.

With some help from some of the maintainers of the module, I was able to use the kernel’s Ftrace functionality to be able to debug without losing the use of my serial TTY. I was then able to rapidly iterate with changes to the kernel module, and load the rebuilt module into the running kernel using rmmod and insmod.

I have submitted my first ever kernel patch⁴, to convert the delay to the high resolution API. This API has a finer timer, allowing the delay to operate much closer to the requested 1 millisecond on retries.

Next Steps

Complete Player Software

With my kernel patch in hand, I can now return to work on the MIDI player software. I want to be able to insert a floppy disk and, with one button, load all the MIDI files stored there into a playlist and began playback. Then I will add buttons for the other basic playback controls.

I’ve been happy with using go-krazy as the Linux distribution. Writing the MIDI player in Go has been pretty nice, and it makes the whole project feel fast.

I’m now battle testing the MIDI parser to make sure my player can withstand whatever is thrown at it. I hope to make this available as open source software by the time I publish the next part of this blog.

Thinking farther into the future, the following project would add a display for album and track metadata.

Custom PCB and Case

I also aim to design a custom PCB for the synthesizer, buttons, display, and RTC, as well as integration with a power supply that can signal the Pi to shutdown before removing power.

I haven’t yet decided if I’ll do this PCB as a custom hat, or, since I’m not using the majority of the hardware functions, as a board housing the Raspberry Pi Compute Module. The decision here affects the case design.

Like the software, I plan on releasing both the PCB and the case designs as

MIDI Albums

To really sell the illusion that, on an alternate timeline for the 90s, we bought our pre-recorded music as MIDI on floppies, I’d love to commission musicians to compose some awesome tracks that can be packaged onto a floppy. Let me know via email if this appeals to you.

Except, perhaps, the floppy drive itself. ↩︎
There are no 12V rails in the current design, as the WaveBlaster-compatible, which we’re able to talk about, doesn’t require it. For compatibility, I’m wanting to include them in the final design. ↩︎
If not Caesar, then perhaps Peter van Houten. ↩︎
At this time, it is still pending review for a future kernel release. ↩︎

Black-box Serial Testing

terin@terinstock.com (Terin Stock) — Fri, 27 Jul 2018 17:43:56 +0000

Occasionally, we all end up working on a program that could, at best, be described as a ball of mud. We want to refactor it into smaller, more testable components, but we don’t want to risk breaking it in the process.

Enter the concept of “black-box testing”. In black-box testing, the functionality of a program is tested by a tester, without the ability to directly leverage implementation details of the program.

For some programs, all we need to do is give it input, and observe the output. As the software grows in complexity, we may have to compare created files, setup mock web servers, and test databases. Many the great author and many the great blog have written on these problems.

The “ball of mud” side project I’m currently working on communicates over a serial port. This presents a larger challenge for a tester: how can we observe behavior of the application with the serial port from our test? Pointing the program at a file won’t work, as it wants to connect to a remote device and configure options, such as baud rate and parity bits.

The long evolutionary history of UNIX provides us with one solution. In the early days of UNIX, the output from the computer was printed onto serially connected teleprinter. When technology marched on, terminal emulators replaced the physical teleprinter, and the functionality needed to create these pseudo-teleprinters was added to the kernel.

As these teleprinters were originally serially connected, their modern virtual forms retain their serial nature.

In our test harness, we can create a new pseudo-teleprinter then pass the emulated end to our program as the serial port. This example uses the Go test harness, but equivalent functionality is available in most programming and scripting languages, so you can use your favorite.

package main

import (
	"io/ioutil"
	"os/exec"
	"testing"

	"github.com/google/go-cmp/cmp"
	"github.com/pkg/term/termios"
)

func TestCLI(t *testing.T) {
	// create and open the pseudo tty master, and the linked
	// child tty.
	pty, tty, err := termios.Pty()
	if err != nil {
		t.Errorf("unable to create pty: %s\n", err)
	}

	// execute our program, passing in the child tty name (eg, /dev/pts/1)
	// as our serial port paramater.
	cmd := exec.Command("ball-of-mud", "-serialport", tty.Name())
	output, err := cmd.Output()
	if err != nil {
		t.Errorf("error running ball-of-mud: %s\n", err)
	}

	// close the child tty. if we forget to do this the
	// master pseudo tty will wait for more data forever.
	tty.Close()

	// read all the written serial data from the master
	// pseudo tty. ReadAll reads until EOF or an error is
	// returned, but Linux sends EIO when attempting to read
	// from a master with no open children. So let's just
	// look for the error string.
	bytes, err := ioutil.ReadAll(pty)
	if err != nil && err.Error() != "read ptm: input/output error" {
		t.Errorf("error reading pty: %s\n", err)
	}

	// confirm we saw the expected text from standard out
	if diff := cmd.Diff("hello", string(output)); diff != "" {
		t.Errorf("unexpected output (-want +got):\n%s", diff)
	}

	// confirm we saw the expected bytes written to the serial device
	if diff := cmd.Diff([]byte{"world"}, bytes); diff != "" {
		t.Errorf("unexpected serial (-want +got):\n%s", diff)
	}
}

As the two parts of the pseudo-teleprinter are connected to each other, writes to one will be readable from the other side. Thus duplex communication is possible for more complex black-box tests.

This method isn’t entirely perfect. For example, on Linux, it’s not always possible to observe the configured baud rate. Even with these limitations, it’s still a helpful tool in our toolkit.

Controller: Extending your K8s cluster

terin@terinstock.com (Terin Stock) — Fri, 04 May 2018 14:45:00 +0200

Building your own Kubernetes Controllers to extend the functionality of your cluster, presented at KubeCon + CloudNativeCon Europe 2018.

Go Internalization and Localization

terin@terinstock.com (Terin Stock) — Wed, 02 May 2018 19:00:00 +0200

An introduction to Go’s text package, which allows developers to build localized applications and services. Presented at Go Cph on May 2, 2018.

Extending Kubernetes

terin@terinstock.com (Terin Stock) — Tue, 01 May 2018 18:50:00 +0200

Talk on extending Kubernetes Dyanmic Admission control with webhooks, presented at GOTO Cph in Copenhagen. This talk was given as being related to our talk at KubeCon + CloudNativeCon Europe.

Game Boy Collage #1

terin@terinstock.com (Terin Stock) — Mon, 13 Nov 2017 05:34:31 +0000

For a few months now, I’ve wanted to work on some new old-school artwork: creating a collage from the internals of a classic Nintendo Game Boy. Similar to the How Things Work series of books, each component of the Game Boy would have annotations and detailed schematics. Unlike those books, the Game Boy would continue to work. After all, if it doesn’t work, it’s not very fun.

8-bit handhelds

Although I’ll only be taking apart one, I have three 8-bit handhelds.

Game Boy (DMG) playing Smurfs

On the left is the original gray Game Boy. It sports a super reflective black and green screen, guzzling 4 AA batteries, and a big boxy shape.

Game Boy Color (GBC) playing Mario Tennis

To the right in the center is a lime green Game Boy Color, capable of up to 32K colors, and backward compatible with the original Game Boy. I have fond memories playing on my lime green Game Boy Color growing up. I can’t even guess how many hours I put on it.

GB Boy Colour playing Mario Tennis

Finally, on the rightmost side, is the GB Boy Colour, a 2010s clone of the Color. The screen is the wrong aspect ratio and the wrong resolution. The custom processor runs the games at the wrong clock speed. The audio hardware, while loud, is a poor emulation of the original. It does, and this is the saving grace, have a backlight LCD screen.

My understanding is that this uses a custom System on Chip design to emulate the Game Boy Color. I haven’t taken it apart, but I think it will be interesting to do so in a future post.

Backside of DMG. Serial Number G27198639

Backside of DMG with the battery compartment visible

Moving back to the original, gray, Game Boy unit, before we take it apart, I should take some pictures of the outside. The serial number and service stickers aren’t in pristine condition, but they’re still in pretty decent shape! I’d like to repair both labels; I’m looking into the best repair methods.

Left side of the DMG

Right side of the DMG

Bottom side of the DMG

The sides look great, too. I’m surprised that the condition is this good. It includes a 3.5mm headphone jack, as electronics should.

Backside of the LCD board in the case

Backside of the main board in the case

After popping it open, I find that it’s relatively dust free. The Game Boy separates into two main pieces, connected by a ribbon cable. The top board contains the LCD screen, input controls, and contrast knob. The bottom board houses the main CPU, coprocessors, audio amplifiers, volume knob, and cartridge loader. Two daughterboards are attached to the bottom board, one for power regulation, the other for the 3.5mm jack.

Front of the main board, along with the power and audio jack boards

The manufacturing date of both boards is July 1992, based on the date codes stamped. That means these capacitors are at least 25 years old. Since most manufacturers rate their capacitors for 15-20 years, I should replace them before this project gets to far along. The boards are otherwise in great shape.

Inside of the front case

Unfortunately, on closer inspection, this unit has two broken standoffs on the front case. I don’t think it’s a problem: you won’t be able to see them in the final design.

DMG's battery contacts are covered in acid

Battery acid left behind in DMG's battery compartment

The biggest problem is the corrosion in the battery compartment. All the terminals were pretty corroded. They were pretty easy to pop out, but lots of battery acid stayed behind in the case.

A quick aside to the kitchen. For Science!

Silly me, I had no terminal cleaner available, nor any other acidic household cleaner. So I improvised a bit: I had an extra lime in the kitchen from a previous meal. Lime juice is a citric acid with a pH about the same as lemon juice and vinegar, so I figured it would work just as well. It’s also fitting, because, you know, both color handhelds are lime green.

No more acid remains

After cleaning, it appears the battery acid etched some of the shininess away. But it otherwise cleaned up nicely. I cleaned up the battery acid in the case using a cotton swab along with the lime juice.

That’s it for now. In the next installment, I will share the plan for how breaking up the Game Boy and wiring it together. I also have ideas for future enhancements I think would be cool. I’ve also ordered a set of replacement capacitors; I’ll do a post on replacing them once they arrive.

Aliasing JavaScript Modules

terin@terinstock.com (Terin Stock) — Tue, 20 Jun 2017 17:45:55 +0000

A number of years ago I wrote a blog post detailing how to replace modules in a Browserify bundle. In the interceding years, the JavaScript ecosystem has had at least three lifetimes, and we’re now due for an update.

Browserify

As the last post focused entirely on Browserify, I’ll first revisit it first. As noted in the previous post, the browserify-swap transform is designed to swap individual files, not modules. We were able to work around it by defining the configuration as a RegExp, but it’s easy to get wrong. You can now use the aliasify transform, which does support replacing modules.

After installing, enable the transform, and add a new section to your package.json.

{
    "browserify": {
        "transform": [
            "aliasify"
        ]
    },
    "aliasify": {
        "aliases": {
            "underscore": "lodash"
        }
    }
}

In addition to aliasing modules, aliasify supports replacments that are relative to the file calling require, as well as using RegExps to define replacement.

Webpack

Webpack supports aliasing modules right out of the box, no plugins needed. Just add an additional section to your Webpack configuration.

{
    "resolve": {
        "alias": {
            "underscore": "lodash"
        }
    }
}

Babel

You can also setup aliases in Babel with the babel-plugin-module-resolver plugin. Like the above, just add a section to your Babel configuration.

{
    "plugins": [
        ["module-resolver", {
            "root": ["."],
            "alias": {
                "underscore": "lodash"
            }
        }]
    ]
}

Yarn

Finally, you can alias before dependencies are even written to disk if you use Yarn¹. Since it breaks compatibility with npm, it’s best if you only use this method in private applications, not public applications or in libraries.

In this method you combine the alias with defining your dependencies in package.json.

{
    "dependencies": {
        "underscore": "npm:lodash"
    }
}

Unlike the other methods in this guide, using the Yarn method also allows you to install the same dependency at different versions by giving them different names.

{
    "dependencies": {
        "lodash3": "npm:lodash@^3",
        "lodash4": "npm:lodash@^4"
    }
}

Unfortunately, it seems this isn’t yet in the documentation. It has, however, been a “Yarn tip”. ↩︎

A rose by any other name…

terin@terinstock.com (Terin Stock) — Wed, 15 Mar 2017 06:10:00 -0700

I gave the following Browserify talk at the Webpack edition of NodeSource’s monthly meetup. Webpack’s Sean Larkin had given his “Core Concepts” talk before I went on, and I made a comparison to Browserify’s concepts.

While my talk is chiefly satirical, I get serious towards the end: Webpack is powerful because of how configurable it is. On the other hand, people like Browserify because it’s opinionated, so you can focus on building your project. I hope to see improvements to Webpack in 2017 that make it more accessible.

TLS with Erlang

terin@terinstock.com (Terin Stock) — Wed, 02 Jul 2014 00:00:00 +0000

I recently configured an HTTP server written in Erlang for secure communication with Transport Layer Security (TLS), successor to Secure Sockets Layer (SSL). Unfortunately, my attempt resulted in TLS errors from both browsers and command line tools. Determined to find a solution, I dug into the HTTP server and Erlang’s SSL library to resolve these TLS connection failures.

In the process I uncovered issues with intermediate bundles and elliptic curve selections, as well as a configuration optimization.

Bundling Intermediate Certificates

When you buy a certificate for your site, you’re likely to also receive an intermediate bundle. This intermediate bundle is a chain of certificates that bind the certificate issued for your site to a trusted certificate located on the user’s computer or browser (a “root certificate”). If this bundle is excluded, the browser won’t trust the connection, since it can’t verify each link of the chain.

The issuer reminds you to add this bundle to your server’s configuration—it is important not to forget this step! Erlang’s SSL library can be configured to include the intermediate bundle by providing the path to the cacertfile option.

ssl:start().
{ok, ListenSocket} = ssl:listen(443, [
    {certfile, "/full/path/to/server_cert.pem"},
    {keyfile, "/full/path/to/server_key.pem"},
    {cacertfile, "/full/path/to/bundle.pem"}
  ]).
ssl:transport_accept(ListenSocket).

Erlang will now send the entire certificate chain to the browser during the connection, and the browser can trust the connection.

Ode to Debugging TLS

At this point, I expected to be done. Unfortunately, while OpenSSL and TLS scanning tools such as sslyze would connect fine, my copies of Chrome, Firefox, and curl refused to connect with cryptic SSL errors such as ERR_SSL_CLIENT_AUTH_SIGNATURE_FAILED and sec_error_invalid_key. Chrome’s debugging logs provided no additional information.

TLS Handshakes: A Primer

To start a connection to a secure site, the client and browser must first configure the secure connection in a process commonly called a “TLS Handshake”.

In a full handshake, two roundtrips between the browser and the server are required:

The browser sends a ClientHello message to the server.

This message includes the highest TLS version supported, lists of supported cipher suites and compression algorithms, and other TLS extensions, including a list of known named elliptic curves.
The server responds with ServerHello, Certificate, an optional ServerKeyExchange message, and ServerHelloDone messages.

The ServerHello message details specific options used for the connections including the TLS version, the cipher suite, the compression algorithm, and any additional TLS extensions. The TLS version should be the highest version support by both client and server. The server maintains a priority list for cipher suites and compression algorithms, and it selects the highest priority supported by the browser.

The server then attaches the entire certificate chain in a Certificate message, so the browser can verify the authenticity of the connection.

If the Certificate message doesn’t contain enough information to allow a client to exchange a session key, such as with a Diffie–Hellman key exchange, then a ServerKeyExchange message is included.

The server then ends with ServerHelloDone.
The client responds with a ClientKeyExchange, ChangeCipherSpec and Finished messages.

The ClientKeyExchange message contains a secret key determined by both sides using public key encryption. The specifics of how the session key is generated is outside the scope of this primer.

With the ChangeCipherSpec message, the browser tells the server to switch to encrypted communication for the rest of the communication.

To verify the integrity of the communications up to this point, a hash of the previous messages is taken and sent as part of the Finished message. The server will also compute a hash over the same messages and compare.
Finally, the server sends ChangeCipherSpec and Finished messages.

The server verifies the hash sent in the client’s Finished message, then acknowledges the encryption communications and finishes, sending a similar hash to the client.

The Failed TLS Handshake

Interested in understanding the exact exchange between my browser and the server, I logged the TLS traffic with the network protocol analyzer Wireshark. The browser and the server successfully exchanged ClientHello, ServerHello, Certificate, ServerKeyExchange, and ServerHelloDone messages before the browser unexpectedly closed the connection.

Inspecting the ServerHello message informed me the server agreed on using TLS 1.2 and choose the TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA cipher suite. Both supported by the browser.

From RFC4492, when the ECDHE_ECDSA, ECDHE_RSA, or ECDH_anon ciphers are chosen, a ServerKeyExchange message is sent containing the ECDHE public key used to derive the shared key.

The ServerKeyExchange portion of the TLS handshake from the server to the client is replicated below.

0000   0c 00 01 49 03 00 16 41 04 b2 33 23 71 c9 da 80
0010   94 d3 ec eb 05 9f e5 36 91 a7 e2 e5 40 78 aa 03
0020   38 4f eb 7c 36 1b 92 21 58 cf c3 e5 b7 08 40 5a
0030   6a eb d2 6a 22 90 e0 47 28 ce 70 9b bb 87 17 d3
0040   4a bc 7c 78 14 ef 97 0d 0d 02 01 01 00 91 7e 3c
0050   ce 9f 06 1d 00 47 4f 53 85 df 2e 04 31 9a 14 a3
0060   25 bd 51 b3 1a 0f dd 3b c3 f4 25 b0 23 d5 34 0a
0070   a3 fc 2a e2 08 34 29 87 00 91 0e 10 6a 40 b3 b5
0080   61 0c 77 9a 8a 0c 50 dc 78 57 ab 2a 51 66 d0 0d
0090   b7 3c 4d c0 28 b9 06 b4 f5 f6 48 f6 5a 02 c2 7e
00a0   8f b2 ac 4b 03 3a 40 c0 e2 c6 2f 77 61 58 ea 0d
00b0   ab 6c 7f 57 be e1 03 0b c6 1e 2a b0 67 ab c2 db
00c0   0a 5b c4 ab 51 9a 76 e6 75 2d e6 ca ce 06 4b f5
00d0   8f dc f0 c1 42 65 14 c0 79 80 51 f2 68 3b 4a 51
00e0   0d 50 5a 01 32 e3 5c 8d cd 8c ec c1 c4 fa 84 3a
00f0   33 37 4c 9d d5 54 f9 6c aa b8 27 27 7b 4a 7c 33
0100   27 8e 48 48 33 87 73 11 9b 92 0b e3 99 49 23 7b
0110   c5 ab 53 ef f2 86 df 56 e5 97 6b 2d 93 5f c0 8a
0120   e6 68 4f 6b 3a 1b 55 26 08 aa c0 36 74 21 ed cc
0130   0e c9 22 0b 97 51 c1 01 48 3f 01 d2 74 fe 36 18
0140   5f 5c 91 47 b3 19 1c 00 69 7f 17 1b c3

0c indicates this message is a ServerKeyExchange message
00 01 49 is the length of 0x000149 (in decimal, 329) bytes
03 is the elliptic curve type, in this case “named_curve”
00 16 is the named curve, secp256k1
The ECDHE public key and signature follows.

At this point the browser verifies the signature and retrieves the elliptic curve parameters and ECDHE public key from the ServerKeyExchange message.

Section 5.4 of RFC4492 ends with the following note:

A possible reason for a fatal handshake failure is that the client’s capabilities for handling elliptic curves and point formats are exceeded

While Erlang supports all 25 elliptic curves named in RFC4492, common browsers only support a smaller subset of two or three.¹ In the above snippet, we see that Erlang chose secp256k1, the elliptic curve used in Bitcoin, and not one supported by my browsers.

Erlang’s early support of elliptic curves are problematic. When picking an elliptic curve, Erlang does not consider the list of supported curves sent by the browser. This has been resolved with the Erlang R16R03-1 release.

Configuration of TLS and Ciphers

Erlang’s SSL library has defaults for the TLS versions, cipher suites and renegotiation behavior. You may want to change these options for client compatibility and for resiliency to TLS attacks.

ssl:start().
{ok, ListenSocket} = ssl:listen(443, [
    {server_renegotiate, true},
    {versions: [ "tlsv1.1", "tlsv1.2" ]},
    {ciphers: [ "ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA" ]}
  ]).
ssl:transport_accept(ListenSocket).

CloudFlare publishes the cipher suites they use with nginx. You can check the ciphers supported by your Erlang installation by running the following in a erl session.

rp(ssl:cipher_suites(openssl)).

I created a patch for CouchDB that adds the configuration options secure_renegotiate, ciphers, and tls_versions to the SSL section:²

[ssl]
certfile           = /full/path/to/server_cert.pem
keyfile            = /full/path/to/server_key.pem
cacertfile         = /full/path/tp/bundle.pem
secure_renegotiate = true
tls_versions       = [ "tlsv1.1", "tlsv1.2" ]
ciphers            = [ "ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA" ]

While presented through the lens of an HTTP server in Erlang, the same basic steps could be extrapolated to any secure server written in any language. Recapping my debugging process: First, I ensured the server is configured to send the entire certificate chain to the client. Then, I tested the connection with a TLS scanner or a network protocol analyzer. Finally, once the server is properly communicating, I took a look at my server’s TLS configuration to ensure it is secure and reflect current best practices.

Edit: An earlier version of this post claimed browsers only supported three elliptic curves: secp192r1, secp224r1, and secp256r1. This information was incorrectly included from an earlier draft. ↩︎
Edit: My patch did not land in CouchDB 1.6.0. CouchDB has since been reorganized into multiple repos, but my patch continues to be in place for a future stable release. ↩︎

Replacing packages in a Browserify bundle

terin@terinstock.com (Terin Stock) — Thu, 27 Feb 2014 00:00:00 +0000

As a developer on a large Backbone application built with Browserify, there are a number of occasions where I want to replace one dependency with another. In this specific case, I wanted to swap underscore for lodash.

Browserify already supports this with the “browser field” in package.json.

There is a special “browser” field you can set in your package.json on a per-module basis to override file resolution for browser-specific versions of files.

This only works for resolution within your package, if any of your dependency packages require Underscore they’ll get Underscore. This is to help ensure your replacements don’t break your dependencies.

However, it’s suboptimal to ship both Lo-Dash and Underscore, as is maintaining a fork simply to replace the dependency. In these edge cases, it’s useful to replace files or packages even within dependencies.

Luckily, the Browserify transform browserify-swap allows you swap dependencies in certain packages, as defined via the @packages key, while generating the output bundle.

As I want to replace Underscore in Backbone, Marionnette and related packages, the configuration seemed pretty straight-forward.

/* package.json */
{
  "browserify": {
    "transform": [
      "browserify-swap"
    ]
  },
  "browserify-swap": {
    "@packages": [
      "backbone",
      "marionette",
      "backbone.babysitter",
      "backbone.wreqr"
    ],
    "all": {
      "underscore.js$": "lodash"
    }
  }
}

I was a bit discouraged to find that Underscore was still present in the output bundle. After triple-checking that my configuration was valid, I broke out the node debugger to find what was wrong.

I believed browserify-swap to swap files while resolving the require calls. The transform actually checks if the current file path matches a RegEx defined in the package.json file and replaces the contents to require the swapped in file.

The Solution

With this information in hand, it became clear that we needed to swap in the underscore package.

/* package.json */
{
  "browserify": {
    "transform": [
      "browserify-swap"
    ]
  },
  "browserify-swap": {
    "@packages": [
      "underscore"
    ],
    "all": {
      "underscore.js$": "lodash"
    }
  }
}

This causes the swap to happen for each instance of Underscore in the bundle, but only the one instance of Lo-Dash would be included.

I believe that the transform should also allow swapping packages for other packages, as the folder structure of a package is usually not part of the public API. I’ve opened an issue against the project.