Newer
Older

DN42 network upgrade

Date: 2026-02-20 00:02
Tags: dn42

Since last Black Friday I've had a lot more cheap VPSes in odd locations under my control for planned use in DN42. Now I've actually got around to implementing that, but it required several changes.

With the previous 10 nodes, it was still practical to manage them by running ssh commands in shell loops. But now with 30 or more, I found that I needed more automation. All of them are running Debian 12 or 13, so the commands are consistent enough to be via Python. One has a German locale, because I didn't bother to learn how to change it, and that changes some of the command output. Maybe next time I'll learn Puppet.

It was straightforward, if tedious, to generate an SSH private key for each node, enter all the connection info into a JSON file, and write some scripts to set them up.

My first real problem was that I needed a way to select OSPF link costs. Previously, they were set manually based on rough geographic distance — 1 on links between nodes in the same location, 100 between nodes in the same general region, 1000 between nodes on different continents. I tried giving all links the same cost to see what would happen, but predictably, that worked quite badly, with packets being sent all the way around the world in order to avoid Europe and the USA northeast. In the end I wrote a tool to ping across every link, and set the OSPF cost to the actual measured ping time. I pinged between every pair of nodes before and after setting link costs, and the proper latency-based routing makes several routes significantly faster, of course, although for some reason, a few that were already fast are a bit slower, even if I added a per-hop fixed processing cost.

The worst ping times across the network are between London and Sydney, around 450ms, which is pretty bad by internet standards, but I'm informed on the mailing list that this is acceptable for DN42. Maybe in the future I'll run a full-mesh alongside the geographically inspired mesh, and offer two service classes: "interesting" and "fast".

As well as cheap VPSes there were a few good deals on less-cheap physical servers, which I might use to implement some services. I installed an Ethereum cryptocurrency node (not yet on DN42), and one for Polygon which is still synchronizing after two weeks.

The network map on the dn42 page on this website hasn't been updated yet.