dn42

If you already know about dn42, most of the information on this page will not be news to you.

dn42 is an imitation of the Internet, for the purpose of learning about the internet. Whereas the real Internet consists of high-speed routers connected by fiber-optic cables, DN42 mostly consists of low-cost virtual servers connected by virtual tunnels, and runs on top of the real Internet, so it's very cheap to join and experiment with.
Just like the real internet, every ISP has the chance to run their own network how they see fit. Only the interconnections between networks are standardized.
Here is how mine is built.

Here is how my network was built. I was inspired by burble (link requires dn42 connection) to publish how my network, AS4242421855, is designed.

Monitoring

As the network grew beyond a few nodes and had some invisible downtime it became obvious that it was necessary to have automated monitoring, underlying every other layer.
A single designated node runs Victoria Metrics, without redundancy. I'm mostly interested in the current state of the system and historical data is a curiosity, so this system has no replication or backup. If the monitoring server itself fails, it'll be pretty obvious. If the monitoring server is partitioned off from some nodes, they'll appear to be down, and I think that's okay.

Victoria Metrics' prometheus scraping seems to silently fail on IPv6 addresses, so I wrote a Python script

Nodes

Like most non-trivial "DN42 ISPs" I have a bunch of low-cost virtual servers. A few real servers are also included. They are spread across several continents — I don't run a geographically local network. I believe this is the most common choice.

Almost all run Debian Linux 12 or 13. I always use the network namespace feature of Linux to completely segregate DN42 traffic from real traffic. This maximizes flexibility within DN42 and minimizes interference with any more important networking functionality of the server. I know that it's also possible to do without this, and not all "DN42 ISPs" use such segregation. (To create a Wireguard interface inside a namespace that tunnels through a different namespace, you have to create it in the outer namespace, and then move it to the inner one with ip netns link set <link> netns <inner ns>)

Node monitoring

Links

Wireguard: Almost all links are Wireguard VPN links.

Provider private interconnect: Some of my VPN providers provide a virtual private networks between different nodes in the same location. When this is available I use it, since it behaves closer to a real physical network. These links don't use Wireguard — the virtual network is directly inserted into DN42.

Physical interconnect: None at present

OpenVPN: My personal computer used to connect to the network via OpenVPN, but I have switched this to Wireguard as it's easier to configure, although not as flexible.

"Hardware"

Like most DN42 participants with non-trivial networks, I have many low-cost virtual servers, and a few real servers, spread across several continents, , running bird2 and interconnected with Wireguard virtual links. I don't connect every node to every other node because that makes for a boring network. Instead I use a network shape that makes some rough geographical sense and also respects the traffic limits of the various servers (even though dn42 traffic is unlikely to hit any limit).

All of my nodes run Linux, and dn42 occupies a separate network namespace from the default. This means that dn42 traffic is strictly segregated from all other networking on the device. Some people choose to run dn42 together with the internet, but I don't.

I have scripts to automate

When a hosting provider implements a private network interface between virtual servers of the same customer, I treat that as part of DN42, without encapsulating it in Wireguard. This more closely simulates a real network interface.

I don't have any real networks in the DN42 mesh net. Some hosting providers provide private virtual network interfaces, and I treat those as part of DN42.

Addressing and routing

The network is primarily IPv6. Out of my overall /48 block, each node gets a /56. A few IPv4 addresses are allocated ad-hoc. IPv4 traffic can be carried between IPv4 peers, or between an IPv4 peer and one of my IPv4 addresses, even through IPv6-only nodes. I use OSPFv3 as an IGP, even for IPv4. OSPFv3 is only for IPv6, so each IPv4 address is mapped to an IPv6 address in a reserved /96 address block. OSPFv3 calculates routes to these mapped addresses, and they are reverse-mapped using ospf6to4. IPv4 traceroutes passing through IPv6-only routers see the dummy address 192.0.0.8.

Peering with other networks

OSPF handles routing within my own network (autonomous system) so it appears as a coherent whole from the outside. Routes received from other autonomous systems are handled by BGP (as usual).

BGP is fundamentally a point-to-point protocol that runs separately on each peer connection, not a single distributed system like OSPF. This is why it's used on connections between networks - it allows networks to set rules at their border. This isn't as much of a thing in DN42 as it is on the real Internet. For example, on the real Internet, a connection between an ISP and a large customer would typically be made to only allow routes to that customer's address ranges and ASN. If the customer advertised a route to another ISP, it should be blocked.

The only inviolable rule is that you mustn't cause a loop. This is enforced using the path attribute of each route. It contains the ASNs which the traffic will go through. Networks must reject routes that already contain their own ASNs, except in very unusual situations, and they must add their own ASN.

The same BGP protocol is used inside the network to make each router aware of BGP routes received by other routers from their peers. This is known as IBGP (internal BGP), not to be confused with IGP. Since internal connections do not add to the path attribute, a different method is used to avoid routing information loops. A basic configuration for IBGP is for every router to connect to every other router via IBGP. To avoid routing information loops, when a route is received on a router through IBGP, it does not redistribute it back through IBGP. A slightly more advanced configuration is for every router to connect to a route reflector - a designated router which does redistribute IBGP routes. Loops are avoided because non-reflector nodes only connect to the reflector. It is possible to use multiple route reflectors, for redundancy. In my network I am using two.

All routers (including those without external connections) receive the same external route information from the route reflector. It is important they come to the same conclusions about the best external routes, or there can be a routing loop. This is assured by default as they all run the same protocol, but I don't like the implied brittleness, so I would like to find an alternative in the future. It's possible to avoid any possibility of routing loops if each router tags each externally-destined packet with the specific external route it thinks that packet should take, and then other routers obey that tag. This can be done with MPLS, but running MPLS over Wireguard introduces a lot of overhead. It could also be done with some completely custom protocol.

All of this is a very standard arrangement and many real networks work the same way.

I have future plans to build a virtual internet exchange at Hetzner since traffic inside Hetzner's network is completely free and unlimited.

Geographical network map

Logical network map (easier to read)

Looks a little bit silly in some ways, but remember the point is to create an interesting network, not the most efficient one.