Independent DNS anycast for an accessible .nl domain
The story behind the development and management of our own DNS anycast network
Until 6 years ago, our DNS infrastructure here at SIDN consisted mainly of unicast servers, with the emphasis on the Netherlands. We believed it was a sound arrangement. However, an upturn in the frequency of DDoS attacks on the DNS obliged us to think again. Attacks on the root name servers in 2015 and on Dyn's servers made it clear to the DNS community that the infrastructure everyone supposed to be robust was actually quite vulnerable. Then, early in 2017, the .nl name servers were hit by a DDoS attack. We were able to repel it, but with difficulty.
We now therefore operate a cluster of anycast name servers ourselves, which provide global coverage. This blog post explains how that network came into being.
A changing world
In 2017, we were operating a number of unicast name servers ourselves, and also had a group of local anycast name servers stationed in the networks of various major internet providers. The set-up enabled us to provide an excellent service to the Dutch portion of the internet. We also believed that the set-up offered good protection against DDoS attacks. Our reasoning was that, although the unicast name servers would soon be overwhelmed if they came under attack, we had the local anycast name servers to fall back on. In an attack scenario, the local servers would be able to continue handling queries from our primary client group, Dutch internet users.
However, the arrangements have subsequently proved to be less resilient than we imagined back in 2017. The size of a typical DDoS attack has increased enormously, beyond anything our unicast name servers could handle. In 2017, an attack involving a few hundred gigabits of data per second was still considered quite large. Today, attack volumes are typically measured in terabits. That kind of attack volume growth is too much for a relatively small server operator such as SIDN to cope with. Meanwhile, Dutch ISPs were moving away from data centres, forming partnerships and merging. Ultimately, there were only a handful of providers left in the Netherlands.
That trend can be seen as a response to the consolidation of internet content providers into the 'Big Five': Alphabet (Google), Amazon, Apple, Meta (Facebook) and Microsoft. Because they provide content that everyone wants, the Big Five are able to drive hard bargains on the pricing of internet connections. At the same time, the volumes of data that access providers need to handle are growing all the time. Against that background, the access providers have sought to strengthen their position through consolidation and by charging sizeable forwarding fees. Despite being essential to the process of accessing content, DNS traffic has felt the squeeze in the industry's power games. While content providers and access providers negotiate about 100-gigabit connections, DNS traffic is measured in megabits at most. Access providers therefore have little incentive to realise (expensive) direct connections to our local anycast name servers. Consequently, when the hardware reached the end of its economic life in 2019, we didn't replace it.
Another recent trend has proven to be influential: people are increasingly accessing the internet from mobile devices. What's more, many of the devices they use are made by a couple of the big content providers mentioned above: Apple and Google. Being extremely data-hungry, mobile devices are often configured to use resolvers on those Big Tech companies' own networks. Consequently, about 60 per cent of all queries relating to the .nl domain now appear to originate from North America. The reason being that the sending IP addresses belong to US companies, even though the servers handling them may be in Western Europe. Against that backdrop, it became apparent to us that we needed a network of name servers with global coverage, in order to quickly and efficiently process all the queries from the dominant mega-networks. Tests by SIDN Labs and others made it clear that anycast was the only viable option. The inclusion of a unicast server within the infrastructure had a negative impact, particularly on latency.
Migration to full anycast, stage 1: external capacity
In 2017, following the attacks on the DNS, we decided that the first step in our migration to anycast should be to buy in third-party anycast services. We concluded a new deal with Netnod regarding the anycast service we had been using since 1999. Under the new contract, Netnod agreed to provide anycast capacity not only for .nl, but also for .amsterdam, .politie and .aw. In addition, new deals were struck with CIRA and nic.at. At that time, we were also using the free service provided by ISC (SNS-PB), but it was already known that the service would be withdrawn not long afterwards. By the start of 2018, therefore, we had a third-party network of 4 DNS anycast servers for the TLDs we operated.
Stage 2: creating our own network
In late 2019, we came to the conclusion that, in the interests of retaining and building up SIDN's in-house DNS expertise, it would be better for us to operate our own DNS anycast environment. In 2020, SIDN Labs therefore investigated the options for rolling out a modern DNS anycast platform of our own. Over the course of the year, we identified a number of principles that could form a basis for our own operational platform. For details, see the earlier SIDN Labs blogpost on the subject.
We decided to set up our own DNS anycast platform to operate as ns1.dns.nl. That implied creating a readily extensible global DNS anycast network. To enable us to meet SIDN's requirements for .nl name servers, the bare metal was acquired from Packet (now Equinix). The same equipment was used as for the SIDN Labs testbed.
Physical construction of our DNS anycast network began towards the end of 2021. An in-house DNS team was set up as well, primarily to oversee the creation of our anycast platform, and then to assume responsibility for operating it. The team's activities were organised according to the DevOps model, with a series of two-week sprints designed to deliver results quickly.
Challenges
The project inevitably presented certain challenges. First, there was the organisational challenge of setting up a DNS DevOps team within SIDN. We now have a professional six-person team of BGP/DNS engineers. Various technical challenges had to be overcome as well. We wanted to automate the entire pathway from the construction to the rollout of anycast nodes, by deploying a CI/CD pipeline. The relevant expertise had to be acquired as we went.
GitOps: controls all moving parts
One of the architectural principles we followed was that the code in git (version management software) should be a true reflection of the production servers:
Build, test and roll out a golden image with a CI/CD pipeline
Each modification is assigned a version number/commit
Release advance/reversal enabled
Enforce uniform configuration and patches by means of regular rollouts.
This approach to the rollout of anycast nodes is ideal for the DNS. The DNS is almost stateless, and anycast provides load balancing and fault tolerance. Following rollout, very little configuration management is required, and zone data can be retrieved automatically. What's more, PCAP and other statistics need to be transferred to the control systems quickly.
In the CI/CD pipeline, an image is built in 3 stages, starting with a minimal OS image. The final stage yields a complete name server image, which can then be rolled out. The pipeline includes security scans and functional tests to assure the quality of the end product. To ensure that that image can be used for the rollout, it is flattened and stored in the cloud.
(Click image to enlarge.)
Figure 1 Overview of the CI/CD pipeline for .nl anycast node rollout.
We decided to perform the rollout manually, in order to gain experience of system rollout. The image rollout procedure starts on a bare metal server. During rollout, checks are continually performed, and the results are fed back into the pipeline. Each stage of the rollout is visualised on a timeline. After that, the various zones are loaded (.nl, .aw, etc). Once the timeline indicates that all zones have been loaded, BGP is enabled, thus incorporating the node into the DNS anycast platform. Using the pipeline, we are now able to activate a node anywhere in the world within 10 minutes.
Our DNS anycast network for .nl went live on 11 April 2022.
The future
In the year since, we have continued adding to the platform and improving our DNSSEC environment. For example, changes have been made to NSEC3 for .nl. And we are now preparing to roll over the algorithm from RSA/SHA-256 to ECDSA Curve P-256. Details of that will be shared shortly in a separate blog post. Where the DNS anycast platform is concerned, we want to review our software choices for the various pipeline stages. Something else we'd like to do is investigate the possibility of fully automated node rollout following the creation of a new image. Finally, we want to make more use of statistics in order to further improve the reachability of .nl by rolling out nodes to additional locations around the world. Within CENTR, the umbrella group whose members are mostly European registries, we also plan to explore the viability of ccTLD operators supporting each other by exchanging services such as our anycast network.
