Raphael Kabo

Navigating git diffs in VSCode with VSCodeVim

2026-02-26T00:00:00Z

In Vim, ]c and [c jump to the next and previous git diff (c is for change). Unfortunately this is a keybinding I have memorized and cannot live without. The venerable Vim plugin for VSCode doesn't support this out of the box, but I managed to wire it up myself - it just took a bit more fiddling than I was expecting.

First I added this to settings.json:

"vim.normalModeKeyBindingsNonRecursive": [
    {
        "before": ["]", "c"],
        "commands": ["workbench.action.editor.nextChange"]
    },
    {
        "before": ["[", "c"],
        "commands": ["workbench.action.editor.previousChange"]
    }
]

This works everywhere except for VSCode's diff viewer, which is sort of cursed. In the diff viewer, VSCodeVim just printed the command name at the bottom of the screen instead of executing it. I spent a while confused by this until Claude suggested that I open the Command Palette and search for "Go to Next Change" while in a diff editor. From here, I can open the little cog (I had never known what the little cog does) and look at the command being executed, which was workbench.action.compareEditor.nextChange, not workbench.action.editor.nextChange. Because VSCode is secretly terrible, this config isn't obviously available anywhere.

Since ]c and [c are already being captured by VSCodeVim, I couldn't bind the diff view versions in settings.json the same way. Instead, these worked when added to keybindings.json:

{
    "key": "] c",
    "command": "workbench.action.compareEditor.nextChange",
    "when": "textCompareEditorVisible && vim.mode == 'Normal'"
},
{
    "key": "[ c",
    "command": "workbench.action.compareEditor.previousChange",
    "when": "textCompareEditorVisible && vim.mode == 'Normal'"
}

With these two sets of shortcuts in place, I could navigate between diffs in VSCode!

Self-hosting Matrix Synapse with Docker Compose and Caddy

2026-02-25T00:00:00Z

Discord recently announced that they're planning to start requiring all their users to upload photos of their faces or government ID for age verification. Their press release was somewhat incoherent in that they are seemingly trying to prevent their investors from fleeing by assuring them that every user will now need to be age verified, but also trying to prevent their users from fleeing by assuring them that very few users will now need to be age verified. This is the same company, by the way, which was hacked in 2025, resulting in approximately 70,000 government ID photos being leaked from one of their third-party age verification vendors. So now might be a really great time to start considering a non-enshittified, self-hosted alternative to Discord.

The field of alternatives to Discord is... somewhat rough. There's a couple of services which have the vague stink of corporate bullshit about them - Rocket.Chat, Zulip, and Mattermost. I really don't want to host anything that describes itself as "bringing together messaging, voice, video, critical apps, and AI to keep teams connected and missions protected". All these services give me the sense that they're suddenly and coincidentally going to go to a freemium model in the next twelve months.

Then there are a handful of services which seem to have sprung up quite recently, are intentionally designed to copy Discord, and have the vague stink of being written by very small teams of devs who are going to burn out and collapse in the next twelve months: Stoat, Fluxer, and Spacebar. Good luck to them all.

And then there's Matrix, which is an encrypted messaging protocol rather than a shiny all-in-one service. The company Element - founded by the same people who created the protocol - also produce user-facing clients, called Element and Element X, but there are other clients, like FluffyChat, and there are a number of protocol server implementations, among them Synapse.

Matrix reminds me a lot of ActivityPub: it's open source, it has a lot of devs, it's got a lot of funding (Element has raised a lot of VC investment money and contributes the vast majority of code to the protocol, but the protocol itself is governed by the independent Matrix.org Foundation, which means the spec can't be rug-pulled by the investors), it has a lot of users, and a lot of them are constantly unhappy with various things that Element does. Development is slow and uneven. Some features get prioritised, while others - for instance a GIF picker widget in Element - languish for years with lots of desperate user requests and little dev interest. It's all very familiar and comforting. All of these, to my mind, are the tell-tale signs of a well-oiled large open source project. So I decided to self-host Matrix for my friends.

Limitations - who is this for?

Like ActivityPub, Matrix has inbuilt federation - that is, users living on one server can join chatrooms and talk to users on other servers. One of the issues with Matrix when self-hosting is that to participate in a federated room, your server has to sync the full state of that room, including all the events and media shared in it. This can be very slow and resource-hungry on lower-end hardware, particularly for large public rooms. Because I only want to host a chatroom for my friends, I've disabled federation from my instance, meaning that all our chats are self-contained and very lightweight.

We also don't really use a lot of Discord features like granular roles, moderation, or intro channels, which Matrix doesn't have mature equivalents for. For what I wanted - a small chat server with a handful of rooms for a handful of friends - Matrix has proven easy, stable, and lean.

One tedious gotcha I've discovered: there appears to be a bug in Element where messages in a room created before a user joins will never appear for that user, even if the room's history visibility is set to allow new members to see old messages. This is disappointing, and there's no fix yet.

Prerequisites

I'm going to assume you already have a home server running Docker Compose with Caddy as a reverse proxy, and a domain name pointed at it. If you don't, I wrote about how I set up my home server a while back and it's still mostly the same setup. I'm going to use matrix.example.com as the domain throughout this guide - replace it with yours.

You'll also need to set up a DNS A record for your Matrix subdomain. If you're using Cloudflare, make sure the proxy is turned off (grey cloud, DNS only) - Cloudflare's proxy is known to cause issues with WebSocket connections, which Matrix clients rely on for real-time sync, apparently resulting in random disconnections that don't occur when connecting DNS-only.

Step 1: Create directories

mkdir -p synapse/data
mkdir -p synapse/postgres

Step 2: Generate the Synapse config

Run a one-off Docker container to generate the homeserver configuration:

docker run -it --rm \
  -v "$(pwd)/synapse/data:/data" \
  -e SYNAPSE_SERVER_NAME=matrix.example.com \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate

This creates synapse/data/homeserver.yaml and a signing key. The server_name is permanent - it can't be changed later without starting from scratch.

Step 3: Edit homeserver.yaml

Open synapse/data/homeserver.yaml. There are a few things to change.

Database

Comment out the default SQLite database and replace it with PostgreSQL:

database:
  name: psycopg2
  args:
    user: synapse
    password: YOUR_STRONG_PASSWORD
    database: synapse
    host: synapse-postgres
    cp_min: 5
    cp_max: 10

Listener

Make sure the listener has x_forwarded: true so that Synapse correctly reads client IPs from behind Caddy. Don't set bind_addresses to 127.0.0.1 - that's for non-Docker installs and will prevent the reverse proxy from reaching the container.

listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false

Disable federation

If, like me, you just want a private server for friends:

federation_domain_whitelist: []

This whitelists no domains, effectively blocking all federation traffic.

Registration

Disable public registration. Set a shared secret you'll use to create accounts from the command line:

enable_registration: false
registration_shared_secret: "GENERATE_WITH_openssl_rand_-hex_32"

Optional tweaks

# Presence broadcasts online/offline/away status to other users in real
# time. It's nice to have but requires a surprising amount of CPU for
# constant status updates - and on a small server, you'll notice who's
# around anyway.
presence:
  enabled: false

# Limit uploads
max_upload_size: 50M

# Automatically and permanently delete media files older than these
# thresholds. The database records are kept (so messages still show
# that media was attached), but the files themselves are gone forever.
# Set these to something you're comfortable with.
media_retention:
  local_media_lifetime: 90d
  remote_media_lifetime: 14d

# Enable link previews. Synapse will fetch a thumbnail and summary for
# URLs shared in chat. The IP blacklist below is required - it prevents
# Synapse from being tricked into fetching URLs on your internal network
# (e.g. a malicious user posting a link to http://192.168.1.1/admin).
# These ranges cover localhost, private networks, and link-local addresses.
url_preview_enabled: true
url_preview_ip_range_blacklist:
  - '127.0.0.0/8'
  - '10.0.0.0/8'
  - '172.16.0.0/12'
  - '192.168.0.0/16'
  - '100.64.0.0/10'
  - '169.254.0.0/16'
  - 'fe80::/10'
  - 'fc00::/7'
  - '::1/128'

For details on all available configuration options, see the Synapse configuration manual.

Step 4: Fix permissions

The Synapse Docker image runs as UID/GID 991 by default, but the generated files are probably owned by root because root ran the setup command:

sudo chown -R 991:991 ./synapse/data/

Step 5: Add to docker-compose.yml

Add these two services to your existing docker-compose.yml (I'm also showing the Caddy service here for context - yours will already exist):

  caddy:
    image: caddy:alpine
    container_name: caddy
    restart: unless-stopped
    volumes:
      - ./caddy/data:/data:rw
      - ./caddy/www:/usr/share/caddy
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
    ports:
      - 80:80
      - 443:443
    extra_hosts:
      - "host.docker.internal=host-gateway"
    networks:
      - caddy_network

  synapse-postgres:
    image: postgres:16-alpine
    container_name: synapse-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: synapse
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: YOUR_STRONG_PASSWORD
      POSTGRES_INITDB_ARGS: "--encoding=UTF8 --locale=C"
    volumes:
      - ./synapse/postgres:/var/lib/postgresql/data
    networks:
      - caddy_network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse"]
      interval: 10s
      timeout: 5s
      retries: 5

  synapse:
    image: matrixdotorg/synapse:latest
    container_name: synapse
    restart: unless-stopped
    depends_on:
      synapse-postgres:
        condition: service_healthy
    volumes:
      - ./synapse/data:/data
    ports:
      - 8008:8008
    networks:
      - caddy_network

The ports: 8008:8008 mapping on the Synapse container isn't strictly necessary for Caddy (which reaches Synapse via the Docker network), but it exposes the port to localhost, which is needed for the register_new_matrix_user command we'll use later to create accounts.

Make sure the POSTGRES_PASSWORD matches what you put in homeserver.yaml.

Step 6: Add to Caddyfile

matrix.example.com {
    reverse_proxy /_matrix/* synapse:8008
    reverse_proxy /_synapse/client/* synapse:8008
}

Caddy handles TLS automatically.

Step 7: Start everything

docker compose up -d synapse-postgres
# Wait a few seconds for Postgres to initialise
docker compose up -d synapse
# Caddy's hot-reload doesn't always pick up changes in a container:
docker compose restart caddy

Check the logs:

docker logs synapse --tail 20

You should see Synapse now listening on TCP port 8008 with no errors. You can verify it's working by visiting https://matrix.example.com/_matrix/client/versions in a browser - it should return a JSON blob of supported spec versions.

Step 8: Create your first user

docker exec -it synapse register_new_matrix_user \
  -c /data/homeserver.yaml \
  http://localhost:8008

Follow the prompts. Make your first user an admin.

Step 9: Sign in

Download Element X on your phone or Element X on your desktop, tap Sign in, change the account provider to https://matrix.example.com, and enter your username and password. You should now have access to your very own, self-hosted, encrypted chat service! Other/older Matrix clients may call this field "homeserver" rather than "account provider" - it's the same thing.

Creating accounts for friends

There's nothing lonelier and more indicative of the self-hosted lifestyle than a beautifully set up service without anybody else on it. Buck the trend. Force your friends to join the revolution. Since public registration is off, you create accounts for them with the same register_new_matrix_user command, then send them their username and password - they can change their password in their account settings afterwards.

More references

The Framework 12 is the Thinkpad X220 we deserve in 2025

2025-07-04T00:00:00Z

Of course I covered it in stickers immediately. 'Trust Your Technolust' is available from my Redbubble store!

Before my Framework 12 arrived in the post today, my favourite ever laptop was a Thinkpad X220 I bought almost 10 years ago, when it was already some 5 years old. I did so many good things on that thing. I wrote a PhD thesis. I coded a utopian social network. I replaced the old barrel charging port with a USB-C port designed by an awesome DIY hacker. I performed open heart surgey on its BIOS chip to replace the boot firmware with Coreboot. I installed at least five Linux distros on it. I took it to hacker camps and festivals. And of course, I covered it with stickers. It was a computer shaped like a friend, and I would trade a hundred beautiful sleek MacBooks for one of those.

The things I loved about the X220 were what everyone loved about it: it was small, built like a cross between a tank and a bento box, had a really excellent keyboard, a great selection of ports (alas, no HDMI), was easy to customize and modify, and parts for repair were cheap and plentiful. But it had its drawbacks too. By 2025, its processor was getting long in the tooth. I could still code on it, but video games were out of the question. Its screen wasn't particularly good, even after I replaced the stock panel with an IPS panel. Its mousepad was beyond terrible - yes, the trackpoint was cool, but it was also the only reasonable way to move a cursor around the screen. It had an awful webcam and I never got around to working out which Bluetooth cards it was compatible with. Its battery life was laughable, even with the extended battery pack with integrated carry handle which made it feel a bit like the world's most Matrix-y handbag. Worst of all, my particular device had a frustatingly hard-to-diagnose bug where the whole thing would just freeze up, with no warning and no pattern, and require a restart to work again. In short, I loved it, but it was time to move on.

When I saw the launch announcement for the Framework 12, I was beyond excited. I didn't own any Framework devices, but I've been following the company for a few years, and this new device, in its product design and its marketing, felt like it could be another computer shaped like a friend. I bought one the hour it launched, ready every one of the steady drip of reviews, and waited very patiently for it to arrive. I've now had it for twelve hours and am writing this blog post on it.

The Framework 12 incorporates the best qualities of the hackable, tactile, and versatile Thinkpad X220 while addressing and solving its drawbacks. It is destined to become my new favourite laptop.

I am impressed with Framework as a company not only because they stick by their customers, are transparent and honest in an era of corporate obfuscation, and prove by their actions their belief that the best way to do business is to let people repair and hack their devices, not lock them in. In the consumer tech space, that's a corporate philosophy every bit as radical as Patagonia's.

I'm actually more impressed by the fact that Framework do all this while their products are consistently misrepresented in the press. Through determination and tenacity - and presumably some heavy investment - Framework are now enough of a contender that their new products are routinely reviewed in major consumer magazines like Wired and The Verge. But the yardstick by which they are reviewed is consistently that of other consumer laptops - which tend to be underpowered, not designed for Linux, barely repairable, and rarely innovative. They also tend to be cheaper. I have for this reason read a number of reviews of the Framework 12 which compare it unfavourably to the MacBook Air and Microsoft Surface, simultaneously lauding its repairability while criticising its price, comparing it to products from companies which make their profits on charging huge sums for repairs or convincing customers to replace perfectly servicable hardware with the next big thing (see, for instance, the incredible marketing boondoggle of the 'AI-ready chip'). Patagonia seems to have sailed through this space - reviews of their products always acknowledge the cost of their products and the production choices they make as a necessary factor of their commitment to helping people and the planet. Framework deserves similar treatment. Their laptops cost what they cost because that's what it takes to create a product built to last and designed for years of future innovation, not just for the current big thing.

So here are my thoughts on the Framework 12, not from the point of view of a consumer magazine, but from that of a coder, hacker, and tinkerer - from the point of someone who thinks hardware design peaked the moment the devices we could replace every part of also got wifi.

The Framework 12 comes in a range of joyful colours, although I did go with light gray for maximum sticker potential. Just as the Thinkpad is synonymous with a slightly soft plastic finish, the finish of the Framework 12 is slighty, delightfully rough and feels tough and sturdy. With its contrasting bumpers and slightly squared off design, and its lovely 12-inch form factor, it's as much of a friendly bento box as the Thinkpad X220, but a little lighter and sleeker. This is not a laptop that will slide out of your fingers - it's a laptop that will stay with you through any adventure.

Unboxing the laptop was an absolute joy, and assembling it was a breeze. I bought a stick of RAM and an SSD online - doing so was significantly cheaper than buying these same third-party items via Framework, something I see The Verge, with its obsession with price, is weirdly quiet on - and both worked without a hitch. I am genuinely astounded by the number of clever, thoughtful technical decisions I've seen inside this one laptop. A keyboard and mousepad top shell that stays attached via magnets and interfaces via pogo pins. Captive screws. Magnetized RAM covers. Modular plug-and-play ports of all varieties - USB-C, USB-A, HDMI, Ethernet, and more - which fit any way you desire into the four port slots on the sides. The freedom of deciding where my USB-C ports will go, and the knowledge that if I'm ever faced with a short HDMI cable I can just switch my HDMI port to the other side of the laptop - these are rare joys that will stay with me for a long time.

I love that the webcam and microphone can both be disabled with a physical switch. I love the touchscreen, which is responsive, really bright, and crisp - very much an improvement on the Thinkpad's. I love the mousepad, which is remarkably good - smooth, comfortable, and precise. I love the keyboard, which has a good deep key travel, full-size keys, and is satisfying and comfortable to hammer away on. Those are the things that make or break devices for me - their tactile and physical instruments. I hated my 'butterfly keyboard and touchbar' MacBook Pro. I loved my Thinkpad. I will love my Framework 12.

I chose to install the Cinnamon Spin of Fedora, a process which also went so smoothly as to be barely worth mentioning. So, an hour after first booting the laptop up, I had a device with a screen which automatically rotated when I put it in tablet mode, with a dismissable onscreen keyboard, connected to my Bluetooth headphones and my local file storage, and I was watching an episode of Star Trek on it - with it propped up in tent mode next to the food processor while I was making dinner. One hour to Star Trek - that's surely the best metric of a convertible, open, flexible, joyful device like this.

I do already have a list of things which could be improved - some in software, some in hardware. When the laptop is in tent mode, I can't use the media keys on the back, even to adjust the volume. This may be controllable by a BIOS setting. The keyboard isn't backlit, which isn't a concern for me because I touch-type. (Ironically, I'm writing in the dark in bed and I misspelled the word 'touch' three times in that sentence). The Cinnamon DE does not present the greatest touchscreen experience - select boxes don't stay open, the global menu isn't properly scrollable, and the onscreen keyboard is hit and miss - but these can all be fixed by contributing to open source code or switching to a different DE, probably Gnome or KDE. The speakers are quite weak - certainly I was having trouble watching Star Trek in a loud kitchen without headphones. But they were fine for background music while I was writing. The screen bezels are relatively wide for a laptop in 2025, but I'm sure that an improved display will be released in a few years. These are all such minor issues and I already love this little thing so much. I've called it Janus, partly because it flips in two directions but also because it looks forwards and backwards - to the repairable, open hardware of the past, and the innovations we haven't even imagined in the future. It's a laptop shaped like a friend, and I can't wait for the adventures we're going to have in the many years we're going to spend together.

Building a Tell-Tale Heart with an Adafruit Prop-Maker FeatherWing

2025-04-13T00:00:00Z

"The Tell-Tale Heart" is a short story by Edgar Allan Poe, in which an extremely sane man very calmly proceeds to explain to you, the reader, how he murdered the old man with whom he was lodging and hid his body under the floorboards. When the police come around, he can still hear the old man's heart beating under the floorboards. It is an excellent short story and if you haven't read it, I highly recommend you go away and do so now.

Illustration by Harry Clarke (1923)

This Halloween, we had friends coming over for a feast, and naturally I decided to build a Tell-Tale Heart of my very own to horrify them with. Come dinnertime, I would dramatically announce that the old man's heart was hidden somewhere in the house, and that the person who found it would get a wonderful prize (candy).

My plan for how the heart should work was as follows:

It would need to be battery powered, so it could beat unattended for a period of time.
It would have to make noise, so it needed a speaker and an amplifier.
It should beat softly, just above the range of normal human hearing, so I could hide it somewhere.
If it was picked up or disturbed - or even if someone moved close to it - it should begin beating much louder.
Once the disturbance ended, it should return to beating softly again.

It was pretty much guaranteed that with behaviour like this, it would upset and disturb my friends thoroughly!

The first thing I did was pick up a squishy heart from Flying Tiger for £5. I have no idea why they made this, but it's hideous and I love it.

Then I took a scalpel (craft knife) to the heart, and removed some of the innards, which were made of fluff. There was definitely enough space inside the heart to hide some electronic components, which convinced me that this project would actually work out.

Then I built a test circuit on a piece of breadboard, using a Challenger RP2040, a MAX98357A amplifier breakout board, and a small speaker from Pimoroni. You can see that in the image at the top of this post.

The next step was to find some heartbeat sounds, for which I went to the incredible Freesound. This involved a lot of boring trial and error, because some heartbeat sounds looked like they might work but came out fuzzy, staticky, or warped from the speaker, and others were mysteriously quiet whatever I did. I referred to the Adafruit docs for my I2S amplifier breakout a lot as I tested. It was particularly helpful that they suggested the WAV files should be converted to mono with 22 KHz sample rate and 16 bits per sample, which I did in Audacity. In the end I actually convered them to 16 KHz, 16 bits per sample, which worked well for the files I chose.

I ended up with two audio files: a slow heartbeat and a fast heartbeat. I popped these on the RP2040 board along with some testing code, and managed to make the speaker play one, then the other.

The next step was to add an acceleremoter into the mix, so the heart could react when it was picked up or sensed a vibration. At this point, I realised that for almost the price of an accelerometer board and the other materials I needed to create a working, battery-powered heart, I could just buy the Adafruit Prop-Maker FeatherWing. This supported a battery and had an onboard accelerometer and amplifier! It's such a versatile board, and I wouldn't hesiate to use one for a similar project again.

Once I got my board, with the full moon glowering through my window and wolves baying in the woods beyond the wild darkness of the unkempt garden, I finally set to work, crafting my monstrous code.

First, we pull in our imports and do some setup. This code will require you to download the extra CircuitPython libraries for the accelerometer (adafruit_lis3dh) and the NeoPixel (neopixel).

import audiobusio
import audiocore
import board
import array
import time
import math
import audiomixer
from audiomp3 import MP3Decoder
from digitalio import DigitalInOut, Direction, Pull
import adafruit_lis3dh
import neopixel

# enable external power pin
# provides power to the external components
external_power = DigitalInOut(board.EXTERNAL_POWER)
external_power.direction = Direction.OUTPUT
external_power.value = True

# Disable NeoPixel
pixel = neopixel.NeoPixel(board.NEOPIXEL, 1)
pixel.brightness = 0

fast_heartbeat_timer = 0
is_fast = False

Then, we set up the onboard amplifier. We fetch the files from the disk, use audiocore to read them into a format the processor recognises, then use audiobusio and audiomixer to create a mixer pointing at the I2S amplifier. Finally, on startup, we send the slow heartbeat noise to the mixer, which means it will begin playing immediately.

# i2s playback
slow_file = open("heartbeat_slow.wav", "rb")
fast_file = open("heartbeat_fast.wav", "rb")
slow_wave = audiocore.WaveFile(slow_file)
fast_wave = audiocore.WaveFile(fast_file)
audio = audiobusio.I2SOut(board.I2S_BIT_CLOCK, board.I2S_WORD_SELECT, board.I2S_DATA)
mixer = audiomixer.Mixer(voice_count=1, sample_rate=16000, channel_count=1,
                         bits_per_sample=16, samples_signed=True)
audio.play(mixer)

# Play slow heartbeat
mixer.voice[0].play(slow_wave, loop=True)
mixer.voice[0].level = 1.0

With our heartbeat playing, the final step is to configure the behaviour which uses the onboard accelerometer. This setup code is taken from the Adafruit documentation.

# onboard LIS3DH
i2c = board.I2C()
int1 = DigitalInOut(board.ACCELEROMETER_INTERRUPT)
lis3dh = adafruit_lis3dh.LIS3DH_I2C(i2c, int1=int1)
lis3dh.range = adafruit_lis3dh.RANGE_2_G

Then we enter an endless loop with while True.

while True:
    if is_fast == True and time.monotonic() >= fast_heartbeat_timer:
        is_fast = False
        mixer.voice[0].play(slow_wave, loop=True)
    if is_fast == False and lis3dh.shake(shake_threshold=10):
        print("SHAKEN!")
        mixer.voice[0].play(fast_wave, loop=True)
        is_fast = True
        fast_heartbeat_timer = time.monotonic() + 10

In the logic inside this loop, which runs every tick, we check if we're beating fast. If we are, we see if the 10 seconds of fast_heartbeat_timer have elapsed. If they have, we unset is_fast and start playing the slow heartbeat again.

In the second part of the logic, we check, instead, if we're beating slowly and we're sensing a vibration. If so, we start beating quickly, set is_fast to True, and set the timer to 10 seconds.

In the final cases (we're beating fast but we haven't waited 10 seconds, or we're beating slowly but we're not sensing a vibration), we just keep looping, waiting for something to happen.

That's all the code our heart needs to do its terrible works! The last bits of the project were to attach a tiny LiPo battery and a speaker to the board, and stuff it inside the soft, protective fluff of the heart.

When my friends found the heart in its hiding place, they were very upset. Some comments included:

"Why is it so squishy"?
"Oh god I can feel the heartbeat coming through the speaker."
"I don't like it, make it stop."
"aaaaaaa"

All in all, a huge Halloween success.

Using Eleventy and S3 to build a personal photo blog

2025-04-13T00:00:00Z

I wanted a nice place to display albums of photos I take (I'm not a great photogapher, but it's nice to be able to share them with friends). I don't use Instagram or Flickr, and websites like 500px seem intimidating. In any case, as soon as I upload a photo to a third party business, I lose control over it. I'd far rather just have a site where I put my photos up and unambiguously license them with a Creative Commons license so anyone can make use of them without a business getting in the way.

Also, I love making websites.

Here's what https://photos.raphaelkabo.com/ looks like with some albums from the last year:

The albums list, which is the home page.

An individual album page. Hi Mahesh!

There's even a lightbox!

I decided to build this site with the static site generator Eleventy, because I really like the way it combines modern JS/TS with the classic concepts (including templates, collections, and layouts) developed by veteran static site generators like Jekyll and Hugo. I could have used Astro, which I had previously used to generate my homepage and for client projects, but I also wanted to learn some new skills - and Astro felt a bit overkill for something as simple as my photo blog concept.

Using S3 for file storage

Planning ahead a little, I didn't want to store a whole bunch of binary files (that is, high-resolution photos) in a version control system like Git. Instead, I decided to store my photos in DigitalOcean's S3-compatible Spaces object storage. This isn't a DigitalOcean sponsored post by any means, but the simplicity of $5/month for 250GB of storage with huge bandwidth limits and a built-in CDN appeals to me. It makes even more sense because I'm using Spaces for a couple of other projects. Once the photos were uploaded to Spaces, my Eleventy site needed to somehow scoop them up, generate thumbnails from them during the build step, add some metadata, and keep track of the high-resolution URLs so I could show them in a lightbox.

Step 1: Set up Eleventy

Eleventy is refreshingly simple to set up. The file system tends to describe the output structure, so with this in my eleventy.config.js, I could write the contents of my homepage, at index.njk, the albums/[album].njk page, and _includes/base.njk:

module.exports = function (eleventyConfig) {
  eleventyConfig.addPassthroughCopy("css");
  eleventyConfig.addPassthroughCopy("favicon.png");

  return {
    dir: {
      input: ".",
      output: "_site",
      includes: "_includes",
      data: "_data",
    },
  };
};

Step 2: Connect to S3

I pull the photos in from S3 as an Eleventy collection. Under _data, I created a YAML database of my albums. This file holds my album metadata, because the S3 directory contains only photos:

// _data/albumMetadata.yaml

albums:
  - folderName: 2024-01-north-finchley
    displayName: North Finchley, London
    date: January 2024
    coverImage: 4
    description: Nikon D600, Nikon 35mm f/2.0 AF-D. Just a soft, quiet winter's day; I'm trying to develop some affection for this area by working on noticing. I think the winter light adds a lovely moodiness to the colours.

  - folderName: 2024-04-around-london
    displayName: Around London
    date: January-April 2024
    coverImage: 6
    description: Nikon EM, 50mm f/1.8, Ilford HP5+ 400 and Kodak Gold 200

  ...

Next, I created an S3 client:

// util/s3Client.js

const { S3 } = require("@aws-sdk/client-s3");
const dotenv = require("dotenv");

dotenv.config();

const s3Client = new S3({
  forcePathStyle: false, // Configures to use subdomain/virtual calling format.
  endpoint: process.env.S3_ENDPOINT,
  region: "us-east-1",
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY,
    secretAccessKey: process.env.S3_SECRET_KEY,
  },
});

module.exports = s3Client;

I adapted this client using the very decent DigitalOcean Spaces documentation.

Finally, we need a file which maps the photos into a format Eleventy recognises, which is at _data/albums.js (this automatically makes the new collection accessible at albums.)

const s3Client = require("../util/s3Client");
const yaml = require("js-yaml");
const fs = require("fs");
const path = require("path");
const dotenv = require("dotenv");

dotenv.config();

const albumMetadata = yaml.load(
  fs.readFileSync(path.join(__dirname, "albumMetadata.yaml"), "utf8")
);

module.exports = async function () {
  const params = {
    Bucket: process.env.S3_BUCKET,
    Delimiter: "/",
  };

  try {
    const listAll = await s3Client.listObjectsV2(params);
    let albums = listAll.CommonPrefixes.map((prefix) => {
      return {
        folderName: prefix.Prefix.slice(0, -1), // Remove trailing slash
        path: prefix.Prefix,
      };
    });

    // Fetch images for each album and merge with metadata
    albums = await Promise.all(
      albums.map(async (album) => {
        const albumParams = {
          Bucket: process.env.S3_BUCKET,
          Prefix: album.path,
        };
        const albumContents = await s3Client.listObjectsV2(albumParams);
        const images = albumContents.Contents.filter((item) =>
          item.Key.endsWith(".jpg")
        ).map((item) => ({
          key: item.Key,
          url: `${process.env.CDN_URL}/${item.Key}`,
        }));

        // Find matching metadata
        const metadata =
          albumMetadata.albums.find(
            (meta) => meta.folderName === album.folderName
          ) || {};

        return {
          ...album,
          ...metadata,
          images,
        };
      })
    );

    return albums.sort((a, b) => b.folderName.localeCompare(a.folderName));
  } catch (error) {
    console.error("Error fetching albums from DigitalOcean Spaces:", error);
    return [];
  }
};

Step 3: Use the collection in Eleventy

Every time Eleventy builds the site or runs in serve mode, it fetches the photos. On the home page, index.njk, I can then run through each album to build a grid:

<ul class="album-list">
  {% for album in albums %}
  <li>
    <a href="./albums/{{ album.folderName | slug }}/">
      // Here I can use album.folderName, album.metadata, and album.images
    </a>
  </li>
  {% endfor %}
</ul>

And on the individual album page, [album].njk, I use Eleventy's amazing pagination feature to pull the collection in directly:

---
pagination:
  data: albums
  size: 1
  alias: album
permalink: "albums/{{ album.folderName | slug }}/"
---

<header class="album-header">
  <h1>{{ album.displayName or album.folderName }}</h1>
</header>

<div class="album-meta">
  {% if album.date %}
  <p class="album-date">{{ album.date }}</p>
  {% endif %} {% if album.description %}
  <p class="album-description">{{ album.description }}</p>
  {% endif %}
</div>

You can see the full source code for this site, including the lightbox JavaScript, on SourceHut.

Notes

Thank you to Mark Llobera for explaining how to display Nunjucks templates in Markdown files without rendering them.

OpenOrb - a curated RSS and Atom feed search engine

2024-04-17T00:00:00Z

Two weeks ago, I built a search engine which only searches over a list of RSS feeds you expressedly give it. If you regularly read ten blogs, you can add those ten to the engine, and then search for stuff in them. I called it OpenOrb - if you click on that link, you'll be able to use my own instance. RSS feed readers can usually do this sort of thing, but RSS readers aren't meant to be shared, so you can think of the search engine as a 'curated feed list as a public service'. I'd love to see people setting up their own OpenOrb instances to allow public access to the lists of blogs and writers they've carefully spent years curating.

I named OpenOrb after the 'pondering my orb' meme, which was originally an illustration by Angus McBride for a 1988 gamebook (a single-player pen-and-paper roleplaying story contained in a single book) called A Spy in Isengard.

Niche, purposefully small-scale, and custom search engines have been turning up in my HackerNews and Mastodon feeds a lot recently. In part, this is a bubble effect - I'm interested in the handmade web and IndieWeb, in self-hosting, and in alternatives to the enshittification of dominant web platforms. When I was getting rid of my Facebook account in 2018-19, I built a small-scale, communally governed social network, Sweet, which is no longer running, and a federated event sharing service, Gathio, which very much is. So I'm exactly the sort of audience to appreciate an alternative search engine.

Google sucks now

Of course, there's another reason why alternative search engines are becoming more popular - it's because Google sucks now. The research paper which that Register article references notes that Google, Bing, and DuckDuckGo - which primarily uses Bing for its search results - all appear to be losing the game against content farms, which exploit the behaviour of search engine algorithms designed to integrate with the SEO economy, flooding search result pages with shitty, machine-generated content. In response, search engine operators have two strategies: (1) constantly screen and adjust search results by removing machine-generated drivel, which costs a lot of money; or (2) take more control of their search results at source - making them less organic, showing more ads, and platforming only the companies who pay them, which makes a lot of money. Whichever strategy Google is leaning into, it's now an adage of sorts that the only way to reliably find a human-generated product review or recommendation on Google these days is by typing in whatever your search term is and adding 'reddit' at the end. Google knows this very well, which is why it's signed a $60 million dollar deal with Reddit to license its user-generated content:

Over the years, we’ve seen that people increasingly use Google to search for helpful content on Reddit to find product recommendations, travel advice and much more. [...] This partnership will facilitate more content-forward displays of Reddit information [...].

The enshittification ouroboros

It's easy to point the finger of blame at the evil, unscrupulous AI herders who have come down from the mountains to graze their flocks of awful, noisy, shitty goats in the pristine meadows of our search pages. It was, of course, Microsoft and Google who gave them the goats in the first place - and created the situation where in the modern Internet economy, if you want to make any money at all, you have to find a way to get your goats into the meadow, any way you can.

But the death knell for the Internet as we knew it wasn't sounded last year, or even in the early 2010s, when content farms - then human-written - became a serious issue. It sounded when the great tech powers decided that access to information was a monetisable concern. As Doctorow writes in the enshittification piece I linked above:

Searching Amazon doesn't produce a list of the products that most closely match your search, it brings up a list of products whose sellers have paid the most to be at the top of that search.

The same goes for Google and Bing. The scale doesn't really matter - it's just particularly appalling when it happens on the titanic level of search engine indexes. On the smallest scale, if I ran a zine distro where people gave me their zines to sell, and I would sell those zines for cost, perhaps with a markup to keep my distro running, every visitor would have the same experience. But if I gave preferential treatment to people who paid me to sell their zine, I'd end up with tables of awful, garbage pamphlets made by people with too much money, and visitors who cared about quality zines would suffer. If I gave even more preferential treatment to producers who spruiked my own zine distro in the pages of their zines, I'd end up with the Google search results page for 'best backpack' in April 2024 - a grim, sad, deeply useless ouroboros constantly shitting its own words back into its mouth.

Everyday In 2024 11 Guide 2024 Backpacks Use To Men Backpacks Best Laptop Of Backpacks 12 Buy in 2024 For For 25 Backpacks The Best Buyer's 2024 Best Best Travel

A comment on a TikTok I saw recently almost hit the nail on the head: "The problem with the Internet is that it always thinks you're shopping, even when you're not". We need to go two steps further: the problem is not with the Internet, it's with the great tech powers which control the flow of easily accessible information on the Internet - Google, Microsoft, Amazon, and Meta. And they don't always think you're shopping. They're desperate for you to always be shopping.

The alternatives

Google's winning formula is scale - that, and it's been around so long that the Internet has been built around its algorithms. Altrernative search engines need to find other niches to work well, and there are plenty. Blogger Seirdy has written a fantastic and comprehensive list of search engines, including some really specific ones. Some of my favourites are:

Stract - open source, "made for hackers and tinkerers", with optional 'optics' written in a simple DSL used to tune search results. For instance, I can search for content only on the Fediverse; only on the IndieWeb and blogs; or only in developer documentation sites.
Marginalia - a precious, lovely search engine which focuses on text-only, non-commercial sites. "Where this search engine really shines is finding small, old and obscure websites about some given topic, perhaps old video games, a mystery, theology, the occult, knitting, computer science, or art."
Teclis - takes a similar approach to Marginalia, and is built by the creator of Kagi, which is a paid-for, ad-free generalist search engine. It combines its indexer with an ad-blocker: if the ad-blocker notices more than 5 ads on the site, it doesn't get indexed.

OpenOrb

It's been a roundabout journey, but we're back at OpenOrb. I mentioned ideas about alternative search engines appearing on my Mastodon feed. I was originally inspired to create OpenOrb by this post by @vesto:

That's precisely what I want OpenOrb to be - a list of human curated websites with search engines restricted to their contents. Gone is the awful, shitty ouroboros of the late capitalist hellweb. Back is the joy of curation, discovery, collection, and categorization which made the early web so delightful, at least in my memory. Who needs Google? If I need a new backpack, I'll ask my friends who have backpacks. If I want to remember which blogger I follow wrote a really excellent guide to setting up the Linux window manager river, I'll use OpenOrb. Fun fact: Google searching 'river window manager setup guide' doesn't even bring up Leon's excellent post.

The current incarnation of OpenOrb works well enough for two day's worth of code, but I've got some future plans for it already:

Better indexing

Currenty, OpenOrb crawls an Atom or RSS feed. The implementation of these feeds is very loose in the wild - some feeds contain a site's entire content, while others will only have the last ten posts. Some will contain full text content for a post, while others will contain only a short excerpt, or nothing. Some will have a reliable 'last updated' date, some will not, or the date will be configured wrong. The way OpenOrb works now, it will index all the content from an RSS feed it's given. If a feed entry doesn't have any content, it will go to the original post that entry links to, and try to read the content from there. But as @vesto suggested on Mastodon, there's another option: properly crawl the content of all the sites imported into OpenOrb (while respecting robots.txt, of course), so that all content, even content not in the RSS feed, will be indexed. This is the dream.

An API

An OpenOrb instance should have a machine-accessible API. This isn't hard to do at all.

Federation

It would be lovely if my personal instance could 'follow' other instances I really like. If someone else has a great collection of D&D and tabletop roleplaying blogs which they're indexing via OpenOrb, but I don't want to import them directly into my OpenOrb, I should be able to connect my instance to theirs. Then, someone searching on my instance could choose to only search over local content, or include content from federated peers.

Better search

At the moment, the actual search engine part of OpenOrb almost wholesale borrows its code from Alex Molas's search engine in 80 lines of Python. This search engine is surprisingly robust! But it doesn't handle more advanced search query features like wildcards, phrases, and lemmatization/stemming (which allows a search engine to perhaps include hits for 'computer' and 'compute' when you search for 'computers' and 'learn' and 'learner' when you search for 'learning'). Over small sets of posts, I don't think these features are necessary, but it would certainly be nice to have them.

If you set up your own OpenOrb instance, I'd love to hear about it. Also, tell me about the alternative search engines you love to use, or - looking further - your favourite alternatives to search engines in general. How do you find information, discover wonderful things, and create meaningful communities and connections in the post-shitty Internet?

WS2812 fairy lights with ESPHome and the Pimoroni Plasma Stick

2023-04-14T00:00:00Z

Last month I picked up the Pimoroni Plasma Stick, which is essentially an RP2040-powered Raspberry Pi Pico bolted onto a terminal block and a Qwiic/STEMMA QT connector, to make it easier to connect WS2812/NeoPixel-style individually addressable RGB(W) LED light strips. That's a lot of terminology!

Once the Plasma Stick arrived in the mail, I connected it to a string of WS2812 LEDs, also from Pimoroni. I like this flexible LED wire because it hangs attractively along a wall like fairy lights, although it is quite fragile - I've had one of these short-circuit and die on me.

My goal was to use the Plasma Stick to create a set of fairy lights I could control through Home Assistant, by creating an ESPHome device.

What are Home Assistant and ESPHome? Here's how ESPHome describes itself:

ESPHome is a tool which aims to make managing your ESP boards as simple as possible. It reads in a YAML configuration file and creates custom firmware which it installs on your ESP device. Devices or sensors added in ESPHome’s configuration will automatically show up in Home Assistant’s UI.

In my mind, ESPHome is a toolkit which allows me to transform regular development boards into 'smart home' devices, without locking myself into the ecosystem of any particular smart home vendor.

Home Assistant is a web interface and mobile app which allows me to create dashboards to control my smart devices, with first-class support for ESPHome. It's much more powerful than that - it also allows me to create complex integrations and automations to make our home feel even smarter. But at the most basic level, it's a way to display a web page where I can control my fairy lights.

As the name suggests, ESPHome is originally designed to be used with ESP boards such as the ESP8266 and ESP32. The Plasma Stick, however, is an RP2040 device with a completely different architecture. But it seemed to be the very easiest way to connect RGB lights to a networked development board - so I started working out how to make ESPHome play nicely with it.

Getting started with ESPHome

This was my first foray into ESPHome, so I started by installing the toolkit (which is just a Python package). However, to make it work with our board, we need to do some Weird Hacks. So that the hacks don't interfere with the rest of my machine, I'm going to perform the whole process in a separate directory structure and use a Python virtual environment:

mkdir -p ~/esphome/{venv,devices}

This creates a directory, esphome/, and two subdirectories, esphome/venv and esphome/devices.

Then I create a Python virtual environment, and activate it:

python3 -m venv ~/esphome/venv
source ~/esphome/venv/bin/activate

Now that my virtual environment is activated, which I can confirm by checking that the left of my bash prompt reads '(venv)', I can install esphome:

pip3 install wheel
pip3 install esphome

pip will install its files only inside my local venv folder. This is very good, because in a minute we're going to directly hack those newly installed files.

Creating the ESPHome device

The lovely thing about ESPHome is that its devices are simply YAML files of simple definitions, and the ESPHome toolkit handles the complex stuff for us. In the devices subfolder, create a file called fairy-lights.yaml with the following contents:

esphome:
  name: fairy-lights

rp2040:
  board: rpipicow
  framework:
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  fast_connect: true

logger:
  level: DEBUG

api:

ota:

light:
  - platform: fastled_clockless
    rgb_order: GRB
    chipset: WS2812
    num_leds: 50
    name: "Fairy Lights"
    pin: GPIO15
    effects:
      - addressable_rainbow:
          name: Rainbow
          speed: 5

Then ceate a file called secrets.yaml:

wifi_ssid: Your WiFi SSID
wifi_password: yourwifipassword

The rp2040 directive object tells ESPHome that we're targeting an RP2040 board, specifically an rpipicow (i.e. Raspberry Pi Pico W) one. According to the ESPHome docs, the platform_version directive is necessary for the time being to prevent compilation errors. The wifi directive pulls in its secrets from the secrets.yml file during compilation - you can just hardcode those values directly if you like. The logger directive provides helpful logging; the api directive allows us to access this device from Home Assistant, and ota allows us to update it over WiFi once ESPHome is already installed.

The longest directive is light, of course, and all the documentation for it is here. We're using the fastled_clockless platform for reasons which will soon become apparent, and because it allows us to implement lovely lighting effects. You may need to change the chipset, rgb_order, and num_leds directives to suit your own lights. There's also a rainbow effect, of course, because rainbows are awesome.

In a perfect world, we would now connect our Plasma Stick to our laptop and run this command:

esphome run fairy-lights.yaml

Sadly, that gives us the following output:

This feature is incompatible with RP2040. Please see note on documentation for FastLED.

That's odd - while the NeoPixelBus platform doesn't have support for RP2040, it looks like the FastLED platform does! Why isn't it working in ESPHome?

It turns out that the version of the FastLED library that ESPHome uses is 3.3.2, while we need to use a version with RP2040 compatibility, introduced in this PR. It's been merged into main, but there hasn't been an official release since then, so it's time for Weird Hacks.

Weird Hacks

Open the file ~/esphome/venv/lib/python3.11/site-packages/esphome/components/fastled_base/__init__.py. If your Python version isn't 3.11, your site-packages/ directory will be in a different directory. This file defines ESPHome's relationship with the FastLED library. Delete the following line, replacing it with the line below:

  await light.register_light(var, config)
- cg.add_library("fastled/FastLED", "3.3.2")
+ cg.add_library("", "dev", "https://github.com/FastLED/FastLED#58f9c15")
  return var

We're telling ESPHome, instead of downloading version 3.3.2 from the fastled/FastLED GitHub repo, to instead download the version of the code at commit hash 58f9c15, and to call it 'dev'.

Now we need to tell ESPHome that this version of FastLED is in fact compatible with the RP2040 architecture. In the file ~/esphome/venv/lib/python3.11/site-packages/esphome/components/fastled_clockless/light.py, add the following line:

  esp32_arduino=cv.Version(99, 0, 0),
+ rp2040_arduino=cv.Version(99, 0, 0),
  max_version=True,

After these two changes have been made, you can try running esphome run fairy-lights.yaml once more, with the Plasma Stick plugged into your computer of course, and this time, the toolkit should successfully compile the code for the RP2040. Give it a few minutes, and you should be able to see it appear on your WiFi network, after which you can add it into Home Assistant through the ESPHome integration.

Caveats

It should go without saying that directly hacking ESPHome's interface with a library may not work! In this case, I adopted the necessary changes from an ESPHome PR, which was closed and abandoned by the author with the message:

I deployed this branch on a pico-w and it resulted in undesired behavior. The pico-w was seemingly crashing at random and in general very unstable.

In my experience running this Plasma Stick for a month, I've seen no crashes, except for some strange behaviour after my WiFi booted the decive off the network. Nonetheless, for a better experience, it's definitely worth keeping an eye on ESPHome's releases to find out when (if ever!) FastLED compatibility goes into production, and replace the Weird Hacks edition of esphome in this venv with the new version at that point.

Happy fairy light hacking!

Best-fit D&D damage to dice calculator

2023-02-02T00:00:00Z

Just here for the goods?

Download the damage-to-dice NPM package here, or use the online version below!

Damage

Modifier

I was recently creating a new homebrew monster for a 5th edition D&D campaign I'm running, and I was horrified to discover how complex the monster creation rules in the Dungeon Master's Guide were. My interesting idea for a horrific aberration to give my players weeks of nightmares was immediately bogged down in the minutae of CR balance and stat calculation.

The quick rules in the guide were a bit of an improvement on the main set, but after asking for advice on Mastodon, I was pointed towards Blog of Holding's superb post on simplified monster generation. This was a huge improvement on the original rules, but in an ideal world, I'd be able to generate a monster in under 5 minutes once I have a clear idea in mind.

What slows me down is that for a given CR of monster, both rules produce damage output as flat numbers, not as die rolls. I don't want to be telling my players 'take 16 damage' every round - I want the joy and fear that can only come with rolling dice. But I also didn't want to spend time doing the maths to find out what amount of d4s, d6s, d8s, d10s or d12s would, on average, give me a result of 16. This sounded like a perfect problem for code!

First, let's clearly articulate the problem: we want a 'best fit' calculation. Furthermore, we only want to roll one type of die for a single attack - it's traditional, and less messy. If my intended average damage output is 16, I want to find out what number of one type of die I need to roll to get an average value as close as possible to 16 - whether that's a little below or a little above.

The Dungeon Master's Guide gives us a formula for the average of a given die type: sides / 2 + 0.5. If we divide our total damage by this average, we find out how many dice we need to roll to make the total. If we round this value to the nearest integer (as we can't roll fractions of dice, at least not very successfully), we can also find out what percentage of the total this 'best roll' comes to:

// damage = 16, die average roll = 3.5
const numberOfDice = Math.round(damage / die.averageRoll); // 5
const damageWithDice = numberOfDice * die.averageRoll; // 17.5
const percentage = Math.round((damageWithDice / damage) * 100); // 109%

With d6s, we overshoot our average damage by 109% - not bad as a best fit!

The rest of the code will simply run through the entire set of dice from d4 to d20, comparing results until we find the best fit of all the best fits. We also add in an optional modifier value, because we want to be able to calculate dice numbers for attack descriptions where we know the damage value and an attack modifier.

const diceMap = [
  { die: "d20", averageRoll: 10.5, sides: 20 },
  { die: "d12", averageRoll: 6.5, sides: 12 },
  { die: "d10", averageRoll: 5.5, sides: 10 },
  { die: "d8", averageRoll: 4.5, sides: 8 },
  { die: "d6", averageRoll: 3.5, sides: 6 },
  { die: "d4", averageRoll: 2.5, sides: 4 },
];

const calculateBestFit = (damage, modifier = 0) => {
  // Remove the modifier from the damage - we can't roll the modifier damage,
  // only the unmodified part of it.
  if (modifier > 0) {
    damage -= modifier;
  }
  return diceMap.reduce(
    (lowest, die) => {
      const numberOfDice = Math.round(damage / die.averageRoll);
      const damageWithDice = numberOfDice * die.averageRoll + modifier;
      const percentage = Math.round(
        (damageWithDice / (damage + modifier)) * 100
      );
      // If the percentage is closer to 100% than the previous best fit,
      // use this one. Math.abs() turns a negative value positive.
      if (Math.abs(percentage - 100) < Math.abs(lowest.percentage - 100)) {
        // Assign the new low value to the reducer's accumulator
        return {
          die,
          numberOfDice,
          damage: damageWithDice,
          percentage,
        };
      }
      // Don't change the accumulator
      return lowest;
    },
    { percentage: Infinity }
  );
};

This function will give us a single best fit value (for 16, it's 3d10, incidentally, giving us an average value of 16.5 and a 103% overshoot). But what if we want to see the whole range of best fits, to compare them with each other? For instance, we might prefer to use 6d4s instead, because they deal 15 damage (a 94% undershoot) but have a higher minimum damage value of 6 than the 3 we get from the d10s.

We can do this by running .map() instead of .reduce() over the list of dice.

const calculateAllPossibleRolls = (damage, modifier = 0) => {
  if (modifier > 0) {
    damage -= modifier;
  }
  return diceMap
    .map((die) => {
      const numberOfDice = Math.round(damage / die.averageRoll);
      // Some die types are too small to provide the target value.
      if (numberOfDice === 0) return null;
      const damageWithDice = numberOfDice * die.averageRoll + modifier;
      const percentage = Math.round(
        (damageWithDice / (damage + modifier)) * 100
      );
      return {
        numberOfDice,
        die,
        damage: damageWithDice,
        percentage,
      };
    })
    .filter(Boolean);
};

This code uses the neat filter(Boolean) trick, which removes null values from the final array before it's returned. I prefer it to .filter(v => v) because to my mind, filtering against a Boolean constructor makes more sense than just assuming that .filter() will return a truthy value for an element.

And that's it! With this second function, we can get a full array of all possible best fit die rolls between d4 and d20. I've created an NPM package which wraps these functions in a command line interface. You can install it and run it from your command line, and you too will finally get the full lowdown on all the different ways you can approximate an average of 16 damage with a die roll:

$ npm install --global damage-to-dice
$ damage-to-dice 16
Best fit: 3d10+0 = 16.5
----------------------------------------
All possible rolls:
2d20+0 = 21 (131% of target damage); min: 2, max: 40
2d12+0 = 13 (81% of target damage); min: 2, max: 24
3d10+0 = 16.5 (103% of target damage); min: 3, max: 30
4d8+0 = 18 (113% of target damage); min: 4, max: 32
5d6+0 = 17.5 (109% of target damage); min: 5, max: 30
6d4+0 = 15 (94% of target damage); min: 6, max: 24

Deploying an Astro site through SourceHut

2023-01-31T00:00:00Z

Today I decided to finally try out Astro, a Javascript framework for building websites, by trialling it on my homepage. Suffice to say I liked it so much, and it was so easy to get to grips with, that this site is now built using Astro.^[1]

I was previously less interested in Astro because I thought it was mostly a tool for building websites using an islands architecture, something we're extensively trialling at work. But having read the excellent Astro docs, I realised that Astro websites are "Zero JS, by default", and that islands are simply an optional, albeit useful feature. With Astro under the hood, this website ends up just as small and static as it was when I deployed it using my self-designed static site generator, Orogene.

With the move to a new framework and a new git repository, I also thought to make use of my SourceHut account and set up an automatic deployment script using SourceHut's superb build.sr.ht build system. With a deployment system in place, my workflow will look like the following:

Write a new post, like this one, on my local machine
Commit it to this site's SourceHut repository
SourceHut will spin up a virtual machine, install Astro, build the static site from the latest commit in the repository, and copy the newly minted static site directly to my server - all without my lifting a finger.

In my research, I found a brilliant post which documented a similar process, but for the static site generator Hugo. I've borrowed extensively from that post - mostly for my own reference - the steps needed to create a secure path between SourceHut's build system and my server, which will prevent my server from being compromised if the private key is leaked. I strongly recommend you read the original post for the context.

Create SSH keypair

We'll be creating a user, homepage_deployer, on the remote server, who will be allowed only to upload files to a specific directory. That user needs an SSH key to log in to the server from inside the build job. Create a keypair on your local machine:

ssh-keygen -t ed25519 -f ~/.ssh/homepage_deployer

Copy the contents of the public key, ~/.ssh/homepage_deployer.pub, for the next step.

Server setup

This is all borrowed from sidhion.com:

# On the server:

useradd homepage_deployer
# Lock the password because we won't be using it.
passwd -l homepage_deployer
mkdir -p /home/homepage_deployer/.ssh

# Manually add the contents of homepage_deployer.pub to the authorized_keys file.
vim /home/homepage_deployer/.ssh/authorized_keys

chown -R homepage_deployer:homepage_deployer /home/homepage_deployer

Next, we make use of a script called rrsync, which is a restricted rsync designed for scenarios where a user should only be permitted to rsync in a specific directory on a remote server. The script is inside rsync's directory and needs to be moved to a location on the PATH:

cp /usr/share/doc/rsync/scripts/rrsync /usr/local/bin/rrsync

Reopen homepage_deployer's authorized_keys file and add the following text at the beginning of the line with the public key, separated from the key by a space:

restrict,pty,command="/usr/local/bin/rrsync -wo /var/www/raphaelkabo.com"

restrict prevents all forwarding for this key and restricts tty allocation; pty re-enables tty allocation (since we need one for the script to run); the final option, command=, will simply execute the provided command whenever this key is used for authentication, ignoring any supplied commands. All these options are documented in the sshd manpages.

The rrsync flags tell the script that it is only allowed to write to the directory /var/www/raphaelkabo.com, which is where my homepage lives on the server.

The final step is to ensure that rsync has access to the directory where it needs to work and its parent directory, if need be. I ran into some issues with this: non-fatal errors caused by rsync failing to save modification times caused the SourceHut build script to register a failure. Simply assigning homepage_deployer to the group which owned /var/www/raphaelkabo.com was not enough; I had to make homepage_deployer the owner of /var/www/raphaelkabo.com and all the files inside it:

chown -R homepage_deployer /var/www/raphaelkabo.com
chmod 755 /var/www/raphaelkabo.com

SourceHut

Add the contents of the private key you created, ~/.ssh/homepage_deployer, to SourceHut's secrets page. If this key is leaked, it will only allow the attacker to rsync write into one directory on the server - hardly the end of the world. Make a note of the UUID which SourceHut generates for the secret.

Build manifest

SourceHut's build system is controlled by a build manifest, such as a file called .build.yml located in the root directory of the repository. This is a pleasantly simple way to orchestrate a build task and I am a big fan. When the .build.yml file is commited to the repository, SourceHut immediately starts its first build, and will then trigger a build using the latest contents of .build.yml on every commit.

As easy as the system might be, I did have to run my builds about twelve times before I got this right, just to save you the trouble. Here is my final build manifest to build and deploy an Astro site using pnpm:

image: alpine/edge
packages:
  - rsync
  - nodejs
  - npm
sources:
  - https://git.sr.ht/~lown/homepage
secrets:
  - a5e2fe34-6fce-48d3-be7b-484400a4ab04
tasks:
  - install-build-deps: |
      wget -qO pnpm "https://github.com/pnpm/pnpm/releases/latest/download/pnpm-linuxstatic-x64"
      chmod +x pnpm
      sudo mv pnpm /usr/bin
  - build: |
      cd homepage
      pnpm install
      pnpm astro build
  - deploy: |
      cd homepage
      rsync -rvz -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/a5e2fe34-6fce-48d3-be7b-484400a4ab04" dist/ homepage_deployer@raphaelkabo.com:/

This file instructs SourceHut to do the following things:

Spin up an Alpine Linux virtual machine
Install rsync, nodejs, and npm
Clone the latest from my repository into a folder called homepage in the working directory
Fetch my secret SSH key and set it up
Download the pnpm sources for Alpine Linux and move them into a location on the PATH
Set up and build my website using Astro
Deploy it via rsync to my server.

As the original blog post notes (with my alterations):

Using -o StrictHostKeyChecking=no is required, otherwise the build job will hang because it doesn’t recognize the server to connect to (and will ask for confirmation before proceeding). [...] Also important is the destination of the rsync command: homepage_deployer@raphaelkabo.com:/. Notice how the path only specifies /. This is because back in the server, we told rrsync to use /var/www/raphaelkabo.com as the root path of any file synced with rsync.

Once I got my rsync permissions issues ironed out, this system worked flawlessly for me. I found the SourceHut build system very pleasant and easy to use, and I can now publish to my website with a simple commit, which is lovely. It complements Astro's design philosophy nicely: 'get out of the developer's way as much as possible to let them do work without fiddling with tooling for weeks'.

I still think you should write a static site generator, though. ↩︎

Finding out when the bins need to go out using Python, Docker, and Pushover

2022-05-04T00:00:00Z

When I lived in Australia, the local council would send us a little calendar a few times a year with all the bin collection dates listed on it. It had a magnet on the back, so you could stick it on the fridge. I think it also included some cheery information about how much the local MP was doing and why we should absolutely vote for them in the next election.

Now I live in Cambridge, and the local council here do not send us a little calendar, which is annoying. Instead they have an online tool where you plug in your postcode and it tells you when your collection times will be, which is neat, except I don't want to be standing in the kitchen every Tuesday Googling 'Cambridge bin collection times', only to find out that the collection was actually the previous day. I want to just know. So I decided to code myself a solution.

I was very inspired by Julia Evans's posts on tiny personal programs and how to use undocumented web APIs, and I felt like I had all the tools I needed to make this happen:

An undocumented web API
Python
Docker, which is a utility which lets me run individual scripts as standalone containers, separate from the rest of my operating system. I'm not great at Docker, but there are lots of good tutorials for it. You don't need to use it - there's wasn't really any need to do so here except to avoid cluttering my Raspberry Pi with lots of little scripts. I won't remember how I started them and where to go to fix them if they break. Don't let this happen to you!
A little server (a Raspberry Pi) to run a Docker container
The delightfully simple app Pushover, which has a web API where I can send it a message and it sends that message to my phone as a push notification. Pushover costs $4.99, which was totally worth it to me, but you could adapt code of this sort to email you instead!

The GitHub repository for this app, which I called BinBoop (because it boops you when there's bins), is here.

Hacking an undocumented web API

The first thing I needed was data on when the bins would go out. I knew that the website had the data, and I had a hunch that it received that data after the page loaded, because the URL of the page stayed general but the data appeared after I plugged my postcode in. So I felt like it was a safe bet that I could see the service that the site itself was using, and use it myself. And I was right!

In Firefox developer tools, I opened up the 'Network' tab to spy on requests as I performed the postcode-inputting ritual. I filtered it to only show XHR requests, which are the asynchronous requests for data a page makes after it's finished loading itself. And lo and behold, I saw two GET requests. The first was to list all the street addresses for the postcode I entered, along with their location IDs in the API that the council were using. The second was to get the bin times for one location ID.

Looking in the 'Response' tab for that request, I could see that it had all the data I needed: an array of collection times, and some other information. I now knew the URL I needed to use to get this data, and very luckily for me, it didn't require any kind of authentication, like a cookie. All it needed was a location ID.

Writing a Python program

Time to break out Python! I find Python a good language for this sort of thing because it gets out of the way to let me do things quickly, and it has a great web API handling library. You could write a similar app in any other language!

It helps me define the behaviour of a program before I write it. Here's what I came up with:

This program will check the bin collections API every evening
If there's a bin collection scheduled tomorrow, it'll send me a phone notification

It's not the most involved program! But I already knew these things: I needed a way for my Python program to do something every evening, to check an API, and to send a phone notification.

I started by finding out how I could make Python do something on a schedule. The library I found is called... schedule! It describes itself as 'Python job scheduling for humans', which is good, because that's what I am.

I already had Python installed, so I just needed to add schedule:

pip3 install schedule

I made a new directory and started editing a file inside it:

mkdir binboop
cd binboop
vim binboop.py

Here's what my file looked like after I set up schedule. I also added some simple functions to print out nice logging messages.

# binboop.py

import time
import datetime
import schedule

alert_time = "21:00"

# Helper functions for logging
def get_current_time():
    return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

def log(message):
    print(f"[{get_current_time()}] {message}")

def check_bin_collections():
    log("Checking bin collection times")

schedule.every().day.at(alert_time).do(check_bin_collections)

while True:
    schedule.run_pending()
    time.sleep(1)

This program uses Python f-strings, which are one of my favourite recent Python features.

Next, it was time to actually check my API. But to do that, I needed to store a secret - the location ID of my house. I wanted to upload this program to GitHub, and I didn't want secret codes and tokens to become public! So I borrowed a technique that's used a lot in Node.js programming - an .env file. This is a file that keeps environment variables, and doesn't get uploaded to GitHub.

# .env

LOCATION_ID=1234567890

To read this file, I needed a new library: python-dotenv.

pip3 install python-dotenv

Back in the Python file, I added the import and some more constants:

...
from dotenv import load_dotenv
...
location_id = os.getenv('LOCATION_ID')
number_of_collections = 1
bin_collections_url = f'https://api.example.com/{location_id}/?numberOfCollections={number_of_collections}'

round_types = {
    'ORGANIC': 'garden waste',
    'RECYCLE': 'recycling',
    'DOMESTIC': 'rubbish',
}
...

The round_types constant is a dictionary: a map of keys to values. Here, it will let me convert the API's descriptions of bin collection types, on the left, into something a bit friendlier, on the right.

Now I was ready to try and access the API. But wait! I needed two more libraries to do that - requests to handle web API requests and responses, and python-dateutil to handle dates a bit better than the standard Python date library! Back to pip:

pip3 install requests python-dateutil

Finally, I added the call to requests to my function:

...
import requests
from dateutil import parser
...
def check_bin_collections():
    log("Checking bin collection times")
    try:
        # Fetch the API data
        response = requests.get(url=bin_collections_url)
        data = response.json()

        # Parse the data from the next collection into friendly formats
        next_collection = data['collections'][0]
        log(next_collection)
        collection_date = parser.parse(next_collection['date']).date()
        collection_types = [round_types[x] for x in next_collection['roundTypes']]
        collection_types_string = ' and '.join(collection_types)
    except Exception as e:
        log(f"Error getting bin collection times: {e}")
...

I like wrapping my code in try/except blocks, because it means my program won't crash out if something goes wrong. In this case, lots of things can go wrong. The API URL might change, or the data format might change, or I might be disconnected from the internet. The except block will just output a log message in all those cases, and some information about the error so I can work out what went wrong later.

Here, I use requests.get to get the response of the API URL. I parse it as JSON, and then pull the date and collection type out of the first collection in the list (the one with the index 0).

To get the collection time, I used dateutil.parser.parse(), which will try and turn a string into a Python date object. In this case, the date was well formatted, so the parser had no trouble. I only needed the date section of the date, not the time, so I added .date() at the end.

To get the collection types, I use a neat Python feature called list comprehension, which is a great way to turn one list into another list. Here, for every value (x) in the roundTypes list inside next_collection, I return the matching pretty name from the round_types map I created earlier. So a list that looks like this:

['ORGANIC', 'DOMESTIC']

Will turn into this:

['garden waste', 'rubbish']

The next line turns that new list into this:

'garden waste and rubbish'

Much nicer!

Now that my program knew when the next collection would be and what would be in it, I needed to find out if I should be alerted or not. After the existing code, inside the try block, I added the following:

def check_bin_collections():
  ...
        # Check if the collection date is tomorrow (we want to be alerted
        # the night before)
        tomorrow = datetime.date.today() + datetime.timedelta(days=1)
        if collection_date == tomorrow:
            payload = {
                "message": f"Don't forget to put the {collection_types_string} {'bins' if len(collection_types) > 1 else 'bin'} out tonight.",
            }
            send_alert(payload)
  ...

First, I calculate what tomorrow is: it's datetime.date.today() plus a timedelta of days=1. Thanks, Python, that was very logical! And if the collection date is indeed tomorrow, it's time to send myself a notification. I build a new message using an f-string, and call a new function, send_alert().

To send a Pushover alert, I created a new application in Pushover and made a note of my API key and user key. These get added to the .env file:

# .env

...
PUSHOVER_TOKEN=1234567890
PUSHOVER_USER=0987654321

Back in the main Python file, here's the new function, and some more setup constants. I looked at the Pushover API documentation to find out what I needed to do here.

pushover_url = 'https://api.pushover.net/1/messages.json'
pushover_params = {
    "token": os.getenv('PUSHOVER_TOKEN'),
    "user": os.getenv('PUSHOVER_USER'),
}
...
# Send an alert via Pushover
def send_alert(payload):
    log("Sending alert")
    try:
        params = {**pushover_params, **payload}
        response = requests.post(pushover_url, params=params)
    except Exception as e:
        log(f"Error sending alert: {e}")

The params = line combines the default Pushover params, which authenticate my app, and the params sent as the payload to the function, which is currently just the message - but I could add lots of other things to those params if I wanted to! Finally, I use requests.post to send Pushover my message. And with that, my whole program is done!

I tested the program a bunch, to make sure that the scheduler, the API fetching and notification sending all worked as expected. A better way to do this would be to write unit tests for the program, which I might do later!

Creating a Docker container

If I wanted to, I could just use a Linux service to run this Python program whenever my server starts, but I was developing the program on my laptop, which has Python, while the Raspberry Pi does not, so running the program there would also require setting up Python. It was instead time to set up Docker (especially because I already have lots of things running using Docker on that Raspberry Pi).

The first thing I needed to do was find a way to tell Docker which Python libraries to install using pip. The easiest way I know to do this is to make a copy of all the packages I've currently got installed into a file called requirements.txt, and then remove the ones I didn't need for this program.

pip3 freeze > requirements.txt

This creates a line for each package pip has installed. I left only the ones I needed:

# requirements.txt

python-dateutil==2.8.2
python-dotenv==0.20.0
requests==2.27.1
schedule==1.1.0

Then, I created a Dockerfile - a file that tells Docker how to create an image. To do this, I mostly followed these excellent instructions, which luckily came up when I searched for 'Python Docker'.

# Dockerfile

FROM python:alpine
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD [ "python3", "-u", "binboop.py"]

The first line tells Docker that I want to base my image on an existing image: the Docker Python image, specifically its Alpine Linux version, because Alpine Linux is a very lightweight and resource-efficient distro. I got that 'tag' from this page. Next, I switch to a new directory inside the image called /app. I copy the requirements.txt file I just created from my host machine into /app, and run pip3 install with the -r flag to install everything it finds in the requirements file. Then, I copy the rest of the host directory over, and finally I run my program with the command python3 -u binboop.py. The -u flag tells Python to send its output unbuffered, which I discovered I needed, otherwise my lovely logs wouldn't come up when I ran docker logs.

Now that my Dockerfile is done, I could test my brand new image. First I build it:

$ docker build --tag binboop .
Sending build context to Docker daemon  66.05kB
Step 1/6 : FROM python:alpine
alpine: Pulling from library/python
df9b9388f04a: Pull complete
a1ef3e6b7a02: Pull complete
7a687728470e: Pull complete
4ecf30de1710: Pull complete
a1f99e431609: Pull complete
Digest: sha256:21c71a37aaa8b4bec1d07f373c7f32322d935accfeeb0b0860be1df6b194ecb5
Status: Downloaded newer image for python:alpine
 ---> 2c167788a673
Step 2/6 : WORKDIR /app
 ---> Running in 1d49995bd5a2
Removing intermediate container 1d49995bd5a2
 ---> 072a20c39822
Step 3/6 : COPY requirements.txt requirements.txt
 ---> 27b9839e5a13
Step 4/6 : RUN pip3 install -r requirements.txt
 ---> Running in ad82f1e9fcf1
[...]
Removing intermediate container ad82f1e9fcf1
 ---> 5980a8c5c83a
Step 5/6 : COPY . .
 ---> d14c4aa07381
Step 6/6 : CMD [ "python3", "-u", "binboop.py"]
 ---> Running in 2e1fc3abe01f
Removing intermediate container 2e1fc3abe01f
 ---> c5caa674313a
Successfully built c5caa674313a
Successfully tagged binboop:latest

Then I check that it's been successfully created:

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
binboop      latest    c5caa674313a   About a minute ago   59.1MB

Then I run it as a container:

$ docker run binboop
[2022-05-04 21:42:14] BinBoop is running!
[2022-05-04 21:42:14] Sending alert

This command then hangs, because BinBoop runs an endless while True loop, but we have verified that it's working perfectly, so we can stop it and carry on with the final step of this process!

Creating a Docker Compose file

To traditionally run a Docker image as a container, you'd need to write a terminal command, passing in various necessary bits of information. One of those is the environment variables that BinBoop needs to connect to its APIs. Notice how BinBoop ran above, but the environment variables weren't set, so when it came time to check the collection times, BinBoop would fail! In the future, I'll add some code to check that these environment variables are set at startup, which would be a better way to handle this issue.

I don't want to pass environment variables in a command line, so I'm going to use Docker Compose - an extra Docker utility which lets me write out my containers as a YAML file, and then run them with a single simple command. Another benefit of this is that I can define a whole bunch of Docker services in one go, and run them all together, which is precisely what I do on my Raspberry Pi.Let's create docker-compose.yml. I created it in the directory above the binboop directory.

# docker-compose.yml

version: "3.3"

services:
  binboop:
    build:
      context: ./binboop
      dockerfile: Dockerfile
    container_name: binboop
    restart: unless-stopped
    environment:
      - LOCATION_ID=1234567890
      - PUSHOVER_TOKEN=1234567890
      - PUSHOVER_USER=0987654321

The binboop service description is a bit like the Dockerfile description. There's a context, which tells Compose to go into the directory binboop inside the one the file is in, and a dockerfile declaration, which gives it the name of the Dockerfile. Most importantly, the environment array tells Compose the names and values of the environment variables that BinBoop needs to run.

With this file in place, in the same directory, I run docker-compose:

$ docker-compose up -d

The -d flag tells Docker to 'daemonise' my services, or run them in the background and restart them when the system restarts. With this command, Compose will do essentially all the same things as the previous individual docker commands, but it will also pass my environment variables across so that BinBoop works.

Once I had my container running, I could make sure that BinBoop was working correctly by checking its logs:

$ docker logs binboop
[2022-05-04 19:52:41] BinBoop is running!
[2022-05-04 19:52:41] Sending alert
[2022-05-04 21:00:00] Checking bin collection times
[2022-05-04 21:00:02] {'date': '2022-05-04T00:00:00Z', 'roundTypes': ['ORGANIC', 'RECYCLE'], 'slippedCollection': True}

And now I know it's working!

This was a fun way to spend an evening, messing around with APIs, Python, and Docker. I learned more about Docker, which is useful, and I enjoyed breaking down my working in this blog post. For instance, it showed me that I hadn't set up any error handling for situations where the environment variables hadn't been defined!

Suspending on idle on Void Linux using Sway WM

2022-03-16T00:00:00Z

I'm setting up another PC with the delightful Void Linux (my third so far!), and I decided I wanted to set this one up with Sway WM, a Wayland-based window manager. This is my first time using Wayland (I have, if anything, been over-cautious), and it's been a real breeze so far. Despite what look like early teething problems with Sway on Void, as far as I can gather from forums, all I did to get sway up and running was follow this guide, the short version of which is:

sudo xbps-install sway swayidle elogind dmenu foot sudo ln -s /etc/sv/dbus /var/services mkdir -p ~/.config/sway sudo cp /etc/sway/config ~/.config/sway/config

I then restarted, changed Sway's terminal emulator of choice to foot in the config file, and ran sway to start it from the console.

I want my computer's display to shut down after 10 minutes of inactivity, and for the computer to go to sleep a minute thereafter. For this, I'm using the excellent Sway companion tool swayidle.

In my ~/.config/sway/config file, I've edited the existing configuration for swayidle to look like the following:

exec swayidle -w
timeout 600 'swaymsg "output * dpms off"' resume 'swaymsg "output * dpms on"'
timeout 660 'exec zzz'

The resume hook after 'exec zzz' in swayidle's configuration doesn't seem to run on a non-systemd system, so I added a file called 01-turn-screens-on.sh in /etc/zzz.d/resume:

#!/bin/sh swaymsg "output * dpms on"

zzz runs this script after resuming from sleep, which is exactly what we want.

Finally, I allowed my own non-root user to modify /sys/power/state, which is what zzz does under the hood. To do so, I added the following line in /etc/rc.local, which runs on system startup:

chown USERNAME: /sys/power/state

Replace USERNAME with your own username, of course. Alternatively, you could create a group called, for instance, power, add your user to that group, and then assign that group to /sys/power/state on startup.

This was relatively easy once I'd figured out how all the bits of it interacted, and I'm still not sure why the resume hook seems to fire a fter the first line of swayidle but not after the second. Nonetheless, this is a lot of work for what seems like such a simple thing, and that's why we love Linux!

Wiring up Wireguard, Caddy, and Docker on a home server

2022-02-09T00:00:00Z

I wanted to write a little guide, mostly for my own future use, to describe how I set up Wireguard on my home sever. Hopefully this will be of use to someone else!

Why Wireguard?

I run a home server (an old, reliable, much-battered HP Proliant called Ottoline). It runs a few public-facing websites in Docker containers, like a library of plants in our house, a gitea, and a Nextcloud sever. It also runs a bunch of services which I'd like to keep private, like a Glances system monitor and Portainer.

I have a domain name (let's call it ottoline.com) which I've linked up to my home sever via Namecheap's excellent dynamic DNS API. This means that I can access services on my home server via subdomains: https://plants.ottoline.com, https://git.ottoline.com, https://glances.ottoline.com. The subdomains are wired up via a Caddy Docker container - requests come in, are received by Caddy, and are proxied to the internal networks of my Docker containers.

I want to keep these two nice features the same:

Accessing services via subdomain names
Accessing services from outside my home network

But:

I want to make some services to stay private and some to stay public.

Wireguard felt like the perfect solution - running Wireguard in a Docker container would allow me to set up a backdoor to my server, accessible only when I'm connected to the Wireguard VPN. Public services would always work, but private services would return a 403 error if the request to them came from outside my home network (and, by extension, my VPN).

Sounds simple! But how do I do it?

Setting up Caddy

I started by reworking my Caddyfile, Caddy's main configuration file.

For publicly accessible sites, the configuration remained the same:

plants.ottoline.com {
  file_server
  root * /usr/share/caddy
  encode gzip
}

nextcloud.ottoline.com {
  reverse_proxy nextcloud:80
}

For private sites, I used a named matcher to match only internal IPs (including Docker's 172.x.x.x IPs):

portainer.ottoline.com {
  @internal {
    remote_ip 192.168.0.0/16 172.0.0.0/8
  }
  handle @internal {
    reverse_proxy portainer:9000
    reverse_proxy portainer:8000
  }
  respond 403
}

If the IP is in the @internal range, it'll be reverse proxied. Otherwise, Caddy responds with a 403. Nice!

With this setup in place, I was already half the way there - I could access public sites from the wider Internet, and private sites only from inside my home network. Now for the VPN.

Setting up Wireguard

I added the following section to my docker-compose.yml and ran docker-compose up -d:

wireguard:
  image: linuxserver/wireguard
  container_name: wireguard
  cap_add:
    - NET_ADMIN
    - SYS_MODULE
  environment:
    - PUID=1000
    - GUID=1000
    - TZ=Europe/London
    - SERVERURL=wireguard.ottoline.com
    - PEERS=2
    - PEERDNS=auto
    - INTERNAL_SUBNET=10.10.10.0
  volumes:
    - ./wireguard:/config
    - /lib/modules:/lib/modules
  ports:
    - 51820:51820/udp
  sysctls:
    - net.ipv4.conf.all.src_valid_mark=1
  restart: unless-stopped

I set up two peers - one for my laptop, one for my phone. When this configuration is run, it will create two peer configuration files, peer1.conf and peer2.conf, inside the ./wireguard directory on the host system. I copied the first one to /etc/wireguard/wg0.conf on my laptop, but I changed the following lines:

[Interface]
...
DNS = 192.168.0.4 # My home server's IP address

[Peer]
...
AllowedIPs = 192.168.0.0/16

The AllowedIPs directive means that when Wireguard is enabled on my laptop, it will only send requests to the local server (which includes the subdomain names I set up) through the VPN, and pass everything else outside the VPN. The DNS directive means all DNS queries will go through the home server, which is essential because Caddy needs to know which requests are internal and which are external.

For my phone, I ran the following command on the server to display a QR code and scanned it with the excellently no-nonsense Wireguard app:

docker exec -it wireguard /app/show-peer 2

I then changed the same fields on the phone as I did on my laptop.

Setting up DNS

The final step in the process is setting up DNS on the home server to forward DNS queries being sent via Wireguard to the right Caddy routes. I used a lightweight service called dnsmasq for this. There seem to be a lot of dnsmasq settings one can set, but these worked for me.

First, I disabled the default systemd-resolved service:

systemctl mask systemd-resolved
systemctl stop systemd-resolved
systemctl disable systemd-resolved

Then I installed dnsmasq and edited /etc/dnsmasq.conf:

domain-needed
bogus-priv
cache-size=1000
server=8.8.8.8
server=4.4.4.4
address=/.ottoline.com/192.168.0.4

This sets two default DNS servers (8.8.8.8 and 4.4.4.4) for unresolved queries, and catches all queries to ottoline.com and its subdomains to the home server's local IP.

Then, turn on dnsmasq:

systemctl enable dnsmasq
systemctl start dnsmasq

Finally, I needed to edit /etc/resolv.conf (I'm not sure why, but considering the rest of this somehow worked, I'm not counting my blessings). I think that without it, DNS queries from inside the home server don't resolve). Because on Ubuntu, resolv.conf is controlled by NetworkManager, it's just a symlink and is reset on reboots:

ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 32 Jan 11 10:47 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

I removed the symlink by renaming /etc/resolv.conf to /etc/resolv.conf.bak and creating a new, regular /etc/resolv.conf, containing the following:

nameserver 8.8.8.8
nameserver 8.8.4.4

With this final piece in place, I am able to access some of my containers only via the VPN, and others anytime.

References

uvcdynctrl-udev.log - a 20GB logfile

2021-02-07T00:00:00Z

I logged on to my computer on Monday this week and ran startx as usual, but Xserver crashed, complaining that it couldn't write to /tmp. At first I thought there was an odd permissions issue or some problem with .Xinitrc, but it turns out after running some exploratory du -h commands that my root partition was completely full.

The culprit turned out to be the file /var/log/uvcdynctrl-udev.log, which had ballooned to 20GB in size. The culprit is libwebcam. This is a well-attested problem which apparently still doesn't have a solution:

I didn't have time to work out how to solve it properly, so I set up a simple-as-it-gets crontab:

0 12 * * 1 rm /var/log/uvcdynctrl-udev.log

No dramas so far!

Using a HD44780 LCD display with MicroPython on the Raspberry Pi Pico

2021-02-06T00:00:00Z

I got my Raspberry Pi Pico today (it's now part of our mandatory lockdown ration, right?), and immediately tried to do something vaguely undocumnented with it - namely, run one of those generic HD44780 LCD displays. Mine was a lovely Adafruit one via Pimoroni; thanks for taking all my money Pimoroni! Suckers, what they don't know is that I'm going to keep spending my money there until I'm bankrupt.

There is what looks like a decent C library for the HD44780 and the Pi Pico, called picoLCD, but I wanted to try out MicroPython, which was completely new to me. Luckily, wjdp has written a perfectly functional generic MicroPython library for the HD44780 and the PyBoard, called micropython-lcd. It won't work on the Pi Pico out of the box, but it took me all of fifteen minutes to adapt it to the Pi Pico. It's pretty straightforward, here's the complete code.

90% of this is not mine - I just adjusted some function calls!

from machine import Pin, Timer
import utime

# Almost entirely copied from micropython-lcd by wjdp
# https://github.com/wjdp/micropython-lcd
class LCD(object):
    # Pinout, change within or outside class for your use case
    PINS = [16, 17, 18, 19, 20, 21]
    # Pin names, don't change
    PIN_NAMES = ['RS','E','D4','D5','D6','D7']

    # Dict of pins
    pins = {}

    # Pin mode, push-pull control
    PIN_MODE = Pin.OUT

    # Define some device constants
    LCD_WIDTH = 16    # Maximum characters per line
    # Designation of T/F for character and command modes
    LCD_CHR = True
    LCD_CMD = False

    LINES = {
        0: 0x80, # LCD RAM address for the 1st line
        1: 0xC0, # LCD RAM address for the 2nd line
        # Add more if desired
    }

    # Timing constants
    E_PULSE = 1
    E_DELAY = 1

    def init(self):
        # Initialise pins
        for pin, pin_name in zip(self.PINS, self.PIN_NAMES):
            self.pins['LCD_'+pin_name] = Pin(pin, self.PIN_MODE)
        # Initialise display
        self.lcd_byte(0x33,self.LCD_CMD)
        self.lcd_byte(0x32,self.LCD_CMD)
        self.lcd_byte(0x28,self.LCD_CMD)
        self.lcd_byte(0x0C,self.LCD_CMD)
        self.lcd_byte(0x06,self.LCD_CMD)
        self.lcd_byte(0x01,self.LCD_CMD)

    def clear(self):
        # Clear the display
        self.lcd_byte(0x01,self.LCD_CMD)

    def set_line(self, line):
        # Set the line that we're going to print to
        self.lcd_byte(self.LINES[line], self.LCD_CMD)

    def set_string(self, message):
        # Pad string out to LCD_WIDTH
        m_length = len(message)
        if m_length < self.LCD_WIDTH:
            short = self.LCD_WIDTH - m_length
            blanks=str()
            for i in range(short):
                blanks+=' '
            message+=blanks
        for i in range(self.LCD_WIDTH):
            self.lcd_byte(ord(message[i]), self.LCD_CHR)

    def lcd_byte(self, bits, mode):
        # Send byte to data pins
        # bits = data
        # mode = True  for character
        #        False for command

        self.pin_action('LCD_RS', mode) # RS

        # High bits
        self.pin_action('LCD_D4', False)
        self.pin_action('LCD_D5', False)
        self.pin_action('LCD_D6', False)
        self.pin_action('LCD_D7', False)
        if bits&0x10==0x10:
            self.pin_action('LCD_D4', True)
        if bits&0x20==0x20:
            self.pin_action('LCD_D5', True)
        if bits&0x40==0x40:
            self.pin_action('LCD_D6', True)
        if bits&0x80==0x80:
            self.pin_action('LCD_D7', True)

        # Toggle 'Enable' pin
        self.udelay(self.E_DELAY)
        self.pin_action('LCD_E', True)
        self.udelay(self.E_PULSE)
        self.pin_action('LCD_E', False)
        self.udelay(self.E_DELAY)

        # Low bits
        self.pin_action('LCD_D4', False)
        self.pin_action('LCD_D5', False)
        self.pin_action('LCD_D6', False)
        self.pin_action('LCD_D7', False)
        if bits&0x01==0x01:
            self.pin_action('LCD_D4', True)
        if bits&0x02==0x02:
            self.pin_action('LCD_D5', True)
        if bits&0x04==0x04:
            self.pin_action('LCD_D6', True)
        if bits&0x08==0x08:
            self.pin_action('LCD_D7', True)

        # Toggle 'Enable' pin
        self.udelay(self.E_DELAY)
        self.pin_action('LCD_E', True)
        self.udelay(self.E_PULSE)
        self.pin_action('LCD_E', False)
        self.udelay(self.E_DELAY)

    def udelay(self, us):
        # Delay by us microseconds, set as function for portability
        utime.sleep_ms(us)

    def pin_action(self, pin, high):
        # Pin high/low functions, set as function for portability
        if high:
            self.pins[pin].value(1)
        else:
            self.pins[pin].value(0)

display = LCD()
display.init()

display.set_line(0)
display.set_string("Hello world!")
display.set_line(1)
display.set_string("I am Pi Pico!")

Here's a photo of the finished monstrosity (wire as I say, not as I do, etc.):

For reference, my wiring is as follows:

HD44780	Destination	Function
1 (Vss)	Ground (0V)	Powers the HD44780's processor
2 (Vdd)	5V	Powers the HD44780's processor
3 (V0)	Ground via a 1K resistor*	LCD contrast control: high for nothing, low for aggressive white squares that hate you
4 (RS)	Pi Pico GP16 (pin 21)	Register Select. High to send data, low to send instructions (like 'clear screen').
5 (R/W)	Ground	Read/write toggle. I've kept it permanently tied low since we only ever output to the display.
6 (E)	Pi Pico GP17 (pin 22)	Enable signal. Makes the thing do the stuff.
11, 12, 13, 14 (DB4-DB7)	Pi Pico GP18, GP19, GP20, GP21 (pins 24-27)	Data pins. We send data through them!
15 (BLA)	5V	Backlight LED anode.
16 (BLK)	Ground (0V)	Backlight LED cathode. I stuck a 220 ohm resistor in here to chill it out a bit.

* My LCD came with a rotary potentiometer to adjust the contrast. The resistor works great too!

If I expand on this, it will be to add custom character functionality, which is something the HD44780 supports. I just need to read the datasheet, on which note, this looks like the one for the specific 16x2 1602A model I'm using. But you might as well just read the Wiki page for the HD44780 - the functionality (like war) doesn't change.

This was a fun thing to mess around with of an evening - now that I'm more familiar with the display, I'm hoping to get it wired up to a Z80 processor I'm currently playing with.

A simple gocryptfs mounting/unmounting script

2020-07-23T00:00:00Z

Nothing fancy, but I got bored of typing out both directories when I was mounting or unmounting my gocryptfs filesystem. This script creates a mount folder, mounts to it, and opens it in Thunar. When you unmount, it also deletes that directory if it's empty. Neat, and makes me happy.

#! /bin/bash

# cryptmount v1.0.0
# mv this script to /usr/local/bin (or similar)
# Change the directories to match yours
# To mount: cryptmount m
# To unmount: cryptmount u

ACTION=$1
ENCDIR=/path/to/encrypted/directory
MOUNTDIR=/path/to/decrypted/directory
FILEMANAGER=thunar

case $ACTION in

  mount | m)
    if [[ $(findmnt -M $MOUNTDIR) ]]; then
      echo "The encrypted directory is already mounted."
    else
      gocryptfs --version | cut -d';' -f1
      mkdir $MOUNTDIR
      gocryptfs $ENCDIR $MOUNTDIR
      $FILEMANAGER $MOUNTDIR
    fi
    ;;

  unmount | umount | u)
    if [[ $(findmnt -M $MOUNTDIR) ]]; then
      umount $MOUNTDIR
      # Make sure the directory is empty before deleting it
      if [ -z "$(ls -A $MOUNTDIR)" ]; then
        rm -rf $MOUNTDIR
      else
        echo "Mount directory not empty, exiting."
      fi
    else
      echo "The encrypted directory is not mounted."
    fi
    ;;

  *)
    echo 'Please specify an action: mount (m) or unmount (u).'
    ;;

esac

Announcing two new online utopian games

2020-06-28T00:00:00Z

Over the last year, I've been really enjoying learning React, and I've recently used all the fun new skills I've learned to code two online card games, both about imagining and exploring utopian spaces and communities, but in very different ways. Both started life as physical games, but have become virtual because of the Covid-19 lockdown.

Loving Allness is a game designed by Sinjin Li which exists as a beautiful physical game with various permutations of rules, and now can also be played online. It is a game about travelling between worlds, creating meanings, and rearranging tiles. It helps tell stories about what we want to take with us, and what we want to leave behind, when we change where we are and what things mean. I refined and simplified some of the rules to make it easier to code, but this was still a serious coding challenge, requiring me to push react-beautiful-dnd to the limits of its capabilities to permit me to rearrange tiles between multiple lists while changing their ownership between different players.

We Have the Square is a game co-designed by me and my utopian collaborator Katie Stone in our combined existence as Utopian Acts. It is a collaborative storytelling game inspired by occupation protest movements, asking players to think through the victories and challenges of creating an oppositional utopian community. While Loving Allness is poetic and evocative, We Have the Square is more open-ended, playing less like a game and more like a storytelling exercise.

Get some friends together and play both of them!

You should write a static site generator

2020-06-18T00:00:00Z

Recently, I decided to learn everyone's new favourite programming language, Rust. I really, really enjoy Rust, mostly because learning it has made me realise that our human brains must do a vast amount of hiding the compile-time errors of our thoughts from us.

My problem was that I am by no means a patient or an exacting coder. I had a brief look at the Rust book, found some source code which looked well-written, hacked together a command-line utility which sort of worked, and then decided that the best way to learn more would be to move this website over from Jekyll to a handmade static site generator. This meant that I spent hours slowly and meticulously adding & to variable names and seeing whether the angry wiggly red lines would move somewhere different instead of, y'know, reading the manual. But hey, many ways to mutate and/or borrow a cat, right?

I don't actually think that using a handmade site generator is particularly brilliant or clever or should be encouraged when amazing open source tools to do this already exist and could probably use the energy of keen programmers to improve their own codebases. However, I think that building a static site generator is a fun learning activity in any language because it taught me about:

Writing, reading, deleting, and copying files
Parsing text and manipulating strings
Parsing command line arguments (I could have also built the generator around a config file, with the same general result)
Writing more efficient code
Error handling (sort of); I mean I ignored most of what I learned but it's now in my brain somewhere

My result is called Orogene. It's named after the characters in N.K Jemisin's astonishing, magisterial Broken Earth trilogy, because I just finished the last book and I feel very emotional. It's messy, panic-prone, and feature-limited, but I'm still really proud of it, and it generated this website. Consider writing your own!

HelloEdit: The demon child of a Markdown editor, Microsoft Word 5.1, and Macintosh System 7

2019-05-29T00:00:00Z

Recently, I booted up Mini vMac, an emulator for classic Macintosh computers from the 1980s, and set up System 7 with Microsoft Word 5.1a, running on a Macintosh Plus. This replicates the first computer I ever used, a retired Macintosh Plus from mum's office. My parents inexplicably set it up in the laundry, and I spent a lot of time writing 'novels' in Word and drawing weird abstract art in MacPaint.

Hello, old friend.

For a certain section of the internet, Word 5.1a is as good as word processors ever got. Indeed, as Kevin Lipe writes in the last article linked there, Word 5.1a embodied many features of Jon Gruber's Markdown - it kept out of the way while typing, it had only the bare minimum of formatting features, and it opens almost instantly. Sadly, these days, you would also need to have set up an emulator or, god forbid, an actual old Macintosh to run it (and getting your files off there is a huge pain).

Or would you?

You may be surprised to discover that I spend a lot of time in the world of Markdown. So I decided to take Kevin Lipe at his very literal word, and create a Markdown editor which looked like Microsoft Word 5.1 running on the glorious 8-bit monochrome display of System 7.

This is the result. It runs on my server, but for personal use I highly recommend you download and set it up locally from the GitHub repository. Files are saved into a very simple database, and the backend itself is an Express app. Overkill, perhaps, but also it took me ten minutes to set up.

HelloEdit is a fully functional wrapper for the lovely Markdown editor EasyMDE by Jeroen Akkerman. I've taken a lot of opinionated liberty with the styling of the Markdown writing area, opting for a starker, less WYSISWYG interface which appeals to my Markdown-writing sensibilities. I drew all the sprites for the buttons and so forth myself - they're mostly direct copies from the original interface, so please don't sue me, retired 1980s Microsoft designers! It was a real joy to re-implement some of these lovely design decisions - I absolutely love the little spark/star sprite that appears when you click the minimise and maximise buttons, and the way the big buttons push down when you click them. Looking at the transition from monochrome to color displays on the old Macintosh computers, I really feel like a lot of the charm of the design was lost when colors were brought in. And that weird purple is kind of unforgivable.

No.

One of the most important part of any UI is the fonts, and I relied entirely on the stellar work of many other people to implement the fonts in HelloEdit. Of course, the person we all have to thank is Susan Kare, one of the original Apple designers, who created iconic fonts like Chicago, Geneva, and Monaco; the infamous Dogcow (Moof!); the Happy Mac and Sad Mac icons; and the Command key symbol. The two main fonts HelloEdit uses, ChiKareGo2 and FindersKeepers, are homages to her work by Giles Booth. For the actual editor, I find it difficult to read the jagged bitmap fonts used in the original Macintosh interface for long stretches of time - and in any case, it's impossible to anti-anti-alias fonts in modern browsers, for good reason. The default font is therefore one of my very favourite sans-serif fonts, Fira Sans, but I also included Fira Mono and three bitmap-ish fonts: TimesNewPixel, Tamzen, and a pixel font I made myself, modelled on New Transport by the amazing Margaret Calvert (appropriately called Calvert). In terms of other design, I lazily relied on FontAwesome to display the interface icons (I think they look great, honestly!).

I hope you enjoy it! I wrote this blog post in it, and I might try to write the remainder of my PhD in it, too...

A script to create print-ready tiny zine layouts on macOS

2019-05-25T00:00:00Z

I recently experienced a headache when I was trying to make a very, very small zine.

Zines are great, and small zines are even better - especially for sharing academic research. People love small things; they're memorable; and the small format forces you to really think hard and critically about the most important things you want to say to an audience. You can make zines and give them out before a panel or conference paper!

By far the easiest way to make a small zine is the fabled one-page zine format. All you need for this is a sheet of paper and a pair of scissors, and it transforms a single A4 page into an adorable 8-page booklet, with space for a secret poster on the back.

But I wasn't content with the 8-page format because it loses the entire second side of the page - I wanted to work out the easiest way to create a zine with regular double-sided pages. My solution requires a stapler and probably a guillotine or paper trimmer, because it involves dividing one sheet of paper, printed on both sides, into eight tiny double-sided sheets. Each group of four sheets is then folded in half, for a total of two tiny adorable 16-page zines from one piece of A4 paper.

It's easy enough to mock a zine like this up in Word - just set your paper size to 74mm wide by 52.5mm tall, with 3mm margins. As far as I know, you unfortunately can't set manual paper sizes in Google Docs.

Important: you need to create exactly 16 pages of zine. The first page is the front cover; the second page is the inside of the front cover, which you can leave blank if you want; the second-last page is therefore the inside back cover; and the last, sixteenth page is the back cover. This gives you 12 inside pages, or 14 if you write on the inside covers.

When you're done writing - 10pt or 9pt font is surprisingly legible if you pick a good sans-serif - save the file as a PDF.

Now comes the difficult part. One of the great benefits of a zine like this is that you can fit two copies of the zine onto one piece of A4 paper, but this requires some truly painful playing around with PDF reordering and n-up layouts. Luckily, the whole point of this blog post is saving you the trouble by automating it for you.

The first thing you need to do is install MacTeX. If you don't use LaTeX and don't fancy downloading a 3GB file, you only actually need the 70MB BasicTeX package, which is available from this page.

Once you've installed MacTeX, if your Terminal is open, close and open it again. Double-check that the necessary components have been installed by running the following two commands in Terminal - both should give you some information rather than an error or a blank line:

pdftex -v
kpsewhich pdfpages.sty

Now you need to install the PDFjam tools - these provide an easy wrapper for the PDF tools that come with pdfTeX. The easiest way to do this is to use a small helper script I wrote - open your Terminal and enter this line:

curl https://raphaelkabo.com/scripts/pdfjaminstall.sh | bash

This script downloads the latest PDFjam source files, extracts them, and installs them in your /usr/bin directory. As ever with strange scripts you find on the internet, you should open the file and read through it to make sure it won't destroy your computer before running the above command.

If you'd rather do it manually, head to the PDFjam page and download the pdfjam_latest.tgz file. Extract it, then copy all the files in the bin/ directory to a directory on your path (like /usr/bin).

Verify PDFjam works by running the following command:

pdfjam --version

Now you're ready to rock! In Terminal, navigate to the directory containing your beautiful 16-page PDF. Enter the following command, substituting the filename of your PDF file:

pdfjam --nup 4x4 --a4paper --landscape INPUTFILE.pdf '8,9,6,11,4,13,2,15,8,9,6,11,4,13,2,15,12,5,10,7,16,1,14,3,12,5,10,7,16,1,14,3' -o OUTPUTFILE.pdf

The file you get out of this looks completely bonkers, but I assure you that once the file has been printed off, cut, and assembled, you will end up with two perfectly logically ordered zines.

Alternatively, you can download a bash script I wrote, available here as a Gist which does this for you. Open the link, save the raw file to your computer, navigate to where you downloaded it, and run it like so (you only need to chmod it once, to make it executable):

chmod +x tinyzine
./tinyzine /path/to/input.pdf

To make it easier to use from anywhere, once you've downloaded it, navigate to the directory it's in and execute the following commands:

chmod +x tinyzine
sudo mv tinyzine /usr/bin/tinyzine

Now you can simply run this command from any directory (like the directory where your PDF file is):

tinyzine input.pdf

It will create a file in the same directory with the appended name '-16-up'.

Now all that's left is to open your file, print it, cut it into four lengthways strips, cut each strip in half, and assemble your tiny zines! Happy zineing!

Introducing DocDown, the easy way to turn academic Markdown into Word documents

2019-01-12T00:00:00Z

In a previous post, I elaborated a complex solution to a problem I was encountering while writing my PhD (which, by the way, I'm still writing - isn't procrastination swell?). I wanted to write in Markdown in the amazing MacOS text editor Ulysses, using citation keys automatically generated by Zotero for my footnotes and bibliography. But I (and my supervisor, and eventually my viva panel) also wanted to see my work in easily readable Word documents, with the footnotes embedded, the citations properly formatted, and any necessary styles integrated.

My process involved creating a little application using Automator to filter Markdown documents through the incredible command line utility Pandoc, a file-converting powerhouse. This application was quite difficult to edit and quite fiddly to set up - and recent changes to the way MacOS handles application accessibility permissions may make it unusable.

So I turned it into a native app (my first ever), called DocDown! I've used Electron, but I haven't yet tried using it on platforms other than macOS. Chances are it won't work on Windows without some modifications. You can download the DMG file here, and visit its GitHub repository here.

DocDown lives in the menu bar, out of your way

DocDown still requires you to provide it with Pandoc, a Zotero .bib file, a CSL file, and a reference docx. Instructions for obtaining all of these are on the original blog post (follow steps 1-10 of the Referencing section and steps 1-4 of the Exporting section - DocDown will handle the rest.)

To make your life easier, the app comes bundled with CSL files for a couple of commonly used citation formats and a couple of basic Word templates I put together - all you need if you use these is your own .bib file.

Note: I don't have an Apple Developer certificate (£79 a year!), so when you first launch DocDown, it will warn you that it's from an unidentified developer. I am the developer, I have identified myself, do not fear. Follow these instructions to open the app - basically, right-click on the app icon and click 'Open' instead of double-clicking.

Enjoy, and let me know on GitHub or by email when you inevitably encounter problems! If anyone would like to help out with the source code on GitHub, your assistance will be warmly welcomed.

Wordpress in a subdirectory with Caddy

2019-01-10T00:00:00Z

I've recently moved my webserver to Caddy, a fantastic modern webserver which is very user-friendly, with a helpful support community, and most importantly, effortless HTTPS on every site through the good folks at Let's Encrypt.

The one issue I had with Caddy was running a Wordpress blog which was located on a subdirectory (https://domain.com/blog) of the domain https://domain.com. Whatever redirect rules I tried, either JS and CSS files wouldn't load, or pretty permalinks (https://domain.com/blog/category-name/) wouldn't work at all. Thanks to the people at the Caddy forums, I finally worked out how to combine redirect rules in my Caddyfile to get everything working.

The important thing to note is that there is no need to create two directives in the Caddyfile, one for the main domain, one for the blog - instead, combine the /blog/ redirect directive directly into the main directive:

domain.com {
	root /var/www/domain.com # Directory on server
	fastcgi / /run/php/php7.2-fpm.sock php # PHP-FPM enabled
	errors /var/log/caddy/domain.com/error.log # Error logging
	# Attempt to rewrite every file under / first to its path, then to a directory, then as a URL query appended to index.php.
	rewrite / {
		to {path} {path}/ /index.php?{query}
	}
	# The same rewrite for Wordpress, with the addition of the /blog/ subdirectory, works flawlessly. Because of the way Caddy's rewrite rules work, any URLs featuring /blog will use this redirect before they attempt the more general rule.
	rewrite /blog {
		to {path} {path}/ /blog/index.php?{query}
	}
	gzip # Compress all files for faster loading.
}

This worked great for me!

Simple image captions in Jekyll

2018-10-08T00:00:00Z

Images, and image captions, are always a problem in Markdown. Not only is there no real way to semantically signify images using a text-only format, but it's never really obvious how to tie image captions to the images to which they belong.

This Stack Overflow post demonstrates the simplest way to display image captions on a Jekyll site or blog while keeping the markup and resulting HTML reasonably semantic. The same process will work for any other site which transforms Markdown into HTML.

Just wrap your caption in any tag, such as an emphasis tag, and put it directly below the image, without a new line. To anyone reading raw Markdown, this will hopefully signify the image and caption are linked. I like using the single asterisk italics tags here and the underscore tags elsewhere, to distinguish actual italicised text from caption text.

![](path_to_image)
_image_caption_

For those seeing the HTML, I style the caption using the following CSS. The + selector will only target an element directly after another, so it doesn't affect any other <em> tags on the page.

img + em {
  font-style: normal;
  font-size: smaller;
}

The CSS stops the caption from being italicised, but if you need to be able to italicise text inside the caption, you could wrap it in bold Markdown tags (**bold text**) and add the following CSS:

img + em > strong {
  font-weight: 400;
  font-style: italic;
}

If you need to make text in a caption both italic and bold, rethink your design choices.

To centre-align captions under images, try this:

img + em {
  display: block;
  text-align: center;
  font-style: normal;
  font-size: smaller;
}

A simple Jekyll rsync script

2018-08-03T00:00:00Z

I've just moved this site to Jekyll, which is the first time I've played around with it properly, and I'm very happy with how easy and logical it has been to use. Thanks, Jekyll.

I keep my Jekyll sites in a folder on my Macbook, and have written this very simple bash script to build and upload any updates to my server. It's a little different to the rsync script available on the Jekyll site because I need it to not delete files and directories I already have on my server when it updates those Jekyll is responsible for. Just change up the bits in all caps to suit you (except the $TARGET variable).

With my configuration, I name my development directory the same as my site URL (raphaelkabo.com, for instance), and then I type ju.sh raphaelkabo.com to invoke it. Remember to pop this script in /usr/local/bin/ or another directory in your path and chmod +x it. For a bit more magic, you can also keep the script in a local directory, like ~/scripts/, then run ln -s ~/scripts/ju.sh /usr/local/bin/ju to be able to invoke the script without typing .sh at the end. Makes you feel like a badass.

#!/bin/sh
TARGET=$1
cd /Users/HOMEDIRECTORY/jekyll-dev-sites/$TARGET/ && jekyll build --incremental && rsync -avh --update /Users/HOMEDIRECTORY/jekyll-dev-sites/$TARGET/_site/ username@REMOTESERVER.COM:/PATH/TO/WWW/DIRECTORY/$TARGET/public_html/

A command line die roller for D&D

2018-08-03T00:00:00Z

Real nerds roll dice from the command line, so I made a command line die roller. It's very basic and doesn't have good error checking, but it's the first proper thing I've ever written in Python. It's heavily adapted from this script, but I've added the ability to repeat a roll multiple times (while losing the ability to roll multiple sets of dice in the same command, which I will bring back eventually).

It's available on Github for all your rolling needs.

My workflow for transforming academic Markdown into beautiful Word documents

2017-05-11T00:00:00Z

12/01/2019

I've bundled a significant part of this setup into a simple menu bar application, available here.

01/03/2018

Updated for compatibility with Pandoc version 2.1.1.

Over the last few months, as I've been working on my PhD, I've set up a workflow to convert my writing (in everyone's favourite simple markup language, Markdown), as if by magic, into Word documents which I can email to my supervisor, without having to explain to her that Markdown is great, really, if you only tried it you'd see. She is busy and does not have time for my shit. As a bonus, this workflow turns all your citations into automatic references according to whatever style you happen to be using (Chicago, MLA, APA, MHRA, anything), with all of the appropriate ibids. and page numbers and everything. Seriously.

To that end, this guide - as much for my own future benefit as anyone else's - lists everything that is required to allow me to transform what is on the left into what is on the right with a click of a goddamn button. We do live in the future.

Writing

First, I write my work in Ulysses. Ulysses is a beautiful Markdown word processor and project manager which has the inner workings of a really powerful writing tool like Scrivener, but stays well out of the way when I write. Ulysses is great and I love it a lot. I've only made minor aesthetic changes to Ulysses' default preferences, as well as one very important non-aesthetic change which I will get to in the next section.

Referencing

When it comes to academic references, I use a reasonably complex system which is, however, extremely flexible, sits in the background, and doesn't disturb anyone. It's also so portable that when I reinstalled macOS recently, it took me fifteen minutes to set it up again. I have named this system Zotero/BetterBibTeX/zotpick-applescript/Service, because that is what it is made of and I'm sorry, I can't be called upon to be imaginative on demand. This is how to set up the same system for yourself:

Use Zotero. If you don't use Zotero, you're bang out of luck, although I know a similar (and slightly more swish) system is already inbuilt for Papers (look, the Ulysses blog even have a tutorial for it - traitors), but I don't want anything to do with Papers because Zotero is beautifully, charmingly ugly, and that's the way I like it. I have tried all of the referencing managers, Zotero is the one for me, I will not apologise. If people want a guide to how to get a lot out use and goodness out of Zotero, please email me and I will comply.
Install Zotero Better BibTeX. Follow the installation instructions here. This is a plugin for Zotero which sets up better (hence the name) citation keys for every item in your database. A citation key is a unique reference to an item - for instance, Judith Butler's 2009 book Frames of War will be given the key butler2009, and when I add Butler's 2009 article 'Performativity, Precarity, and Sexual Politics' to my Zotero database, it will automatically be named butler2009a, to distinguish it from the book. You can also change these citation keys manually to whatever you want!
Once you've installed Better BibTeX, go into Zotero. You need to change a few things. Open Preferences, click the Better BibTeX tab, and then under Citation keys change the citation key format to [auth:lower][year], like this:

Now the citation keys for all your papers are going to look something like this:
Now go to the File menu in Zotero and click Export Library. You are about to create an auto-updating list of all the citation keys in your Zotero library! For Format, select 'Better BibLaTeX'. Don't tick any of the other boxes apart from 'Keep updated'. Select where to save this file - I keep mine in my Dropbox.
Now go back to Preferences - Better BibTeX. Go to the Automatic Export tab. You'll see your Better BibLaTeX library. I changed the Automatic export option here to On change, which means it'll update my library as soon as I change or add anything to Zotero, but you can keep yours however you want. This finishes the Zotero section of the referencing system.
Now download zotpick-applescript. This is a magical piece of code which will allow you to drop your newly minted citation keys into any text processor you happen to be using. You actually only need the file named 'zotpick-pandoc.applescript' - open it up and copy the contents.
Now open Automator (in the Applications folder). Click New Document and select Service. At the top, select no input in any application. Then drag the 'Run AppleScript' module from the left sidebar into the main window. Replace everything in the text box with the contents of the 'zotpick-pandoc.applescript' you copied. Save your new service!
Open System Preferences. Go to Keyboard, then Shortcuts, then Services in the left sidebar, scroll down to the bottom, click on your newly named service, click the 'Add Shortcut' button, and press your chosen shortcut for quickly activating Zotpick. I use ⌘-` (tilde key, on the top left corner of my keyboard).
Now go to your text editor of choice. When you need to add a citation, first type two square brackets and place the cursor between them. Then press your chosen shortcut. The Zotpick window will appear - search for the item you need, select it, add prefixes, suffixes, and page numbers. When you press Enter, the full citation key will be copied into your text editor.
This is where the non-aesthetic change to Ulysses comes in. Ulysses' standard Markdown parser will render [] as the beginning of a link, as would any standard Markdown parser. This doesn't matter one bit for the export stage, but Ulysses will try to dress up your citations as links, which is bothersome. To avoid this, you need to change the link symbols in Ulysses to something else, like angle brackets - to do so, follow the Preparations for Ulysses section in this guide.

Exporting

Now for the fun bit. Before you proceed, make sure you have the following things set up:

A Zotero BibTeX library which stores all your citation keys.
A Markdown-formatted document which incorporates your citation keys in square brackets in the places where your references are going to go (whether you are using inline references or footnotes - I use footnotes, my partner loathes them, but then there's no accounting for classicists - place your citation keys before your full stops. The export engine will sort it all out for you.

Now you are going to install Pandoc. Pandoc is an obscenely powerful text converter which can transform almost any popular file format into almost any other popular file format. The downside - if you don't use the Terminal much - is that it's a command line utility, but we're going to turn that command line utility into a simple app.

To begin, download the file named 'pandoc-2.1.1-macOS.pkg' from this page. Install the package. (Alternatively, if you are comfortable using Homebrew, install Pandoc through Homebrew: brew install pandoc. This will make future updating easier.)
Test that Pandoc has installed correctly. Open Terminal and type the following: pandoc -v. If you get some information about Pandoc's version, it's working. Close the Terminal.
Now you need to set up two things for Pandoc to be able to create a beautifully formatted and referenced Word document. The first of these is your CSL file of choice. There are CSL, or Citation Style Language files for almost any referencing system under the sun, which contain all the necessary rules to automate referencing. Go to the CSL repository and find your favourite style (I use Chicago note and bibliography with no ibid. - yes, you can be that specific with some of the more popular styles). Download this file and save it somewhere you'll want to keep it (I created a 'Scripting' folder in my PhD folder in my Dropbox).
The second thing you need to do is create a reference Word document file - a template with preset styles so Pandoc knows what sort of file to output. This is a finicky process, described in the Pandoc manual (search for 'reference-doc'). Alternatively, download the one I made - it's Quite Nice. You can consult the manual to see which styles you can tweak yourselves. Save it somewhere sensible too.
Now open up your trusty Automator again. This time you want to create an Application.
Drag the module called 'Run Shell Script' (in the Utilities section) into the main window. Make sure that your shell is '/bin/bash' and you are passing input as arguments. Now copy the following piece of code into the text box:
```
title=$(basename "$@" .md)
echo -e "\n\n# Bibliography" >> "$@"
/usr/local/bin/pandoc -s --filter=/usr/local/bin/pandoc-citeproc --bibliography /PATH/TO/BIBTEX/LIBRARY/Zotero.bib --csl /PATH/TO/CSL/FILE/FILENAME.csl --reference-doc /PATH/TO/REFERENCE/DOC/reference.docx -f markdown+smart -t docx -o /PATH/WHERE/YOU/WANT/TO/EXPORT/"$title".docx "$@"
open /PATH/WHERE/YOU/WANT/TO/EXPORT/"$title".docx
```
The first line sets the filename of your new file to be identical to the filename of whatever file you sent into it. The second line puts the heading 'Bibliography' at the end of the document - that way the bibliography Pandoc will automatically create will have a heading. You can delete this line if you don't need it. The third line is the Pandoc command. You need to edit all the paths in ALL CAPS in this code. Make sure you've got the right location for your exported BibTeX Zotero library, your CSL file, your reference.docx, and the place you want your converted file to end up. If you're not sure what the path of a folder is, use this one weird trick: in Finder, grab the blue folder icon at the very top of the window, next to the name of the window, and drag and drop it into the text box. Cool, right? If your path has spaces in it, put it in double quotes, like this: "/Users/bobross/Happy Squirrels/reference.docx".
I then also dragged the 'Display Notification' module into the main Automator window below the script module, so that I'd know when the conversion was done. In practice, it happens pretty much instantaneously, but it's cute. Here's what mine looks like:
Save your Automator application in your Applications folder, because that's where it deserves to go - I've named mine DocDown.
Now for the cool bit. You can simply drag any text file with the extension .md onto the DocDown application and it will do its magic conversion business. But in Ulysses, you can make the magic happen without even leaving the editor. Click on the Share button in the top right, select Text, Markdown, then click the A button and select Other. Find DocDown. Your current Ulysses document will be converted instantly into a Word document in your chosen location, with references flown in from your Zotero library and edited by your CSL file. Well done. You are academia god. We all bow down.