Upgrade to DNSSEC signing process for the .nl zone

The .nl zone now has more than six million domain names, of which more than half are DNSSEC-enabled. That means the domain names' DNS records have digital signatures – cryptographic seals of authenticity – attached to them.

Secure and resilient

All the DNS information needed for web domains and mail addresses below the .nl top-level domain to work properly is published in the.nl zone file. The file is updated every half an hour, so that the latest additions and changes are effective as soon as possible. That involves first generating the generic zone file from the central database in which all the domain information for .nl is recorded. The (redundantly engineered) signing server that runs OpenDNSSEC is then triggered to sign the newly generated zone file. Once that's done, the resulting zone file (complete with public keys and digital signatures) can be distributed to our public DNS servers, where the DNS information is made publicly available. In practice, it is of course your operating system or network's resolver that retrieves the DNS information needed to translate a domain name into an IP address. It's during retrieval of the information that any available DNSSEC signatures are checked. For security reasons, no private keys are held on the signing server itself. They're kept on the Hardware Security Module (HSM), which is also redundantly engineered and has the task of generating the cryptographic signatures. The private keys therefore never need to leave the high-security HSMs. The second signing server is located in a separate data centre and fully synchronised with the first machine. A complete copy of the OpenDNSSEC configuration is therefore available on the second server, enabling us to switch over at short notice in the event of an issue with the first signing server.

Software problems resolved

"We began work on this upgrade exactly a year ago," says SIDN's DNS & Systems Engineer Stefan Ubbink. "The trigger was that NLnet Labs had announced they were going to withdraw support for version 1.4 at the end of October 2020. In the interests of continuity, we have a support contract with NLnet Labs, which has proved useful to us on a number of occasions down the years. When we initiated the upgrade programme, NLnet Labs sent a support team to help us install the new version of OpenDNSSEC in our testing and acceptance environment. We started the rollout with the three other zones that we operate – .amsterdam, .aw and .politie – because, being smaller, they're easier to work with." "One or two issues did crop up, but we worked with NLnet Labs to resolve them. One problem was that the active software's memory use gradually increased over time. Once NLnet Labs had fixed that memory leak, a second minor leak came to light. Another issue was that the connection with the database dropped out from time to time. By helping to resolve those issues, we've made an important contribution to the stability of the software."

Testing

SIDN uses the standard version of OpenDNSSEC, supplemented by an array of scripts responsible for monitoring the signing and publication process and verifying all individual steps and outcomes. "Migration to the new version necessitated revision of the scripts," explains Ubbink. "For example, in version 2, the old policy enforcer has been replaced by a proper daemon, which monitors execution of the various tasks. So we've had to change the arrangements for checking whether the right keys have been used in the zone. We're doing the software build ourselves. That way, we've always got the environment ready to quickly apply patches in an emergency." All told, it took a year to complete the transition, mainly because of the need to proceed very carefully, resolving bugs and revising scripts along the way, and the need to exhaustively test the new servers. "Testing involved setting up the new system to operate in real time, in parallel with the old one, so we could see whether everything was working properly," Ubbink recalls. "That implied waiting a long time for each ZSK pair rollover, because rollovers are only once every ninety days. Despite the prolonged testing in our acceptance environment, it turned out that there was an error in the migration scripts. That caused a recent production incident, which fortunately we were able to resolve quickly with help from NLnet Labs."

New policy enforcer

Of course, the decision to migrate had a lot to do with support for the old version of OpenDNSSEC being withdrawn, but there are also advantages to using version 2.1. "The new policy enforcer is particularly welcome from our viewpoint. The old one had to be woken up once an hour," says Ubbink.

Berry van Halderen, Lead Developer at NLnet Labs, agrees that the updated enforcer is one of the big plusses with version 2. "The software now provides much better support for multiple rollover scenarios and different key types. Double-key and double-signature rollovers are now possible, for example, and you can work with shared and combined keys. Rolling over to another cryptographic signing algorithm is now supported as well. Because the enforcer has a better overview of the signed zone, it's now also safe to change various timing settings when performing a key rollover."

New functionality

"Another advantage is that we can now start talking to NLnet Labs about new functionality," adds Ubbink. "For example, we'd love to see built-in support for high availability (HA) and failover. As things stand, if anything were to go wrong, we'd have to manually switch over to our second signing server. That would take about two hours in total, because of the size of the .nl zone and the need for extreme care. Dialogue about issues and new features is made much easier by the fact that NLnet Labs is based here in the Netherlands and by the longstanding sponsorship relationship we have with them." "Having embedded the successive OpenDNSSEC steps in their own control and verification scripts, SIDN has a highly customised implementation of our software," Van Halderen points out. "The software hasn't been designed with that kind of implementation in mind, but it's fine to do it that way. It also provides us with a reason to think about improving support for such implementations."

Future improvements

Each half-hour zone refresh cycle typically involves 200 to 250 domain names. Although it's tempting to focus exclusively on the small number of new and updated records for those names, Van Halderen sees merit in always generating and processing an entire new zone file, as SIDN does. "OpenDNSSEC can certainly handle deltas, but it's much easier to work with the complete zone file. That way, you always have a comprehensive picture of the state of the zone and you know straight away whether everything's as it should be, without analysing historical data." With a view to facilitating that approach, Van Halderen believes OpenDNSSEC could be improved by reducing the number of times a zone file needs to be translated from ASCII to binary and back again. However, he doesn't think significant performance gains are possible. "From tests, we know that, even if OpenDNSSEC is running on a slow server, 90 per cent of the time required to sign a zone file is down to the HSM."

Other registries

Other registries wanting to start using version 2 of OpenDNSSEC are likely to be the main beneficiaries of the upgrade we've made. "We've got an extremely big zone file that includes a very large number of signed domain names, and that means we are an interesting use case," explains Ubbink. "We know a few other registries that are already using version 2, including one large registry, but they use OpenDNSSEC in a different way." "With this upgrade, we're demonstrating that version 2 of OpenDNSSEC can work well when implemented in this way and on this scale. We've also been able to resolve various issues with the software, meaning that version 2 is ready for immediate deployment, even by large TLD registries. So upgrading is very much in line with SIDN's commitment to playing a pioneering role, especially on security."

Wider improvement programme

Our signing server upgrade is part of a wider infrastructure improvement programme. Other changes we're making include phasing out Ubuntu in favour of Red Hat Enterprise Linux (RHEL) wherever possible. Early next year, we're planning to replace our HSMs as well. Our DNSSEC systems will then be fully up to date. Also in the pipeline is construction of our own anycast infrastructure, probably based on the use of NSD (another NLnet Labs product) and Knot (developed by CZ.NIC) for the public servers.