arvb.net

It always is DNS

5 min read

A few days ago I was Solving Arch WiFi issues in my home network. I still stand by that solution, but I will admit I had a moment of doubt because today my internet connection started to fail. Fortunately (or perhaps unfortunately) it wasn’t just my desktop that had issues: every device in the network was unable to access the web. While frustrating, at least I knew it wasn’t my specific configuration that broke everything.

My first thought was that perhaps we exceeded some sort of ISP data limit, but I quickly discarded that idea since ping 1.1.1.1 worked and the ICMP responses, while slow, did come back:

➤ ping 1.1.1.1 -c 10
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=9.50 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=10.2 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=56 time=29.6 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=56 time=64.7 ms
64 bytes from 1.1.1.1: icmp_seq=6 ttl=56 time=33.4 ms
64 bytes from 1.1.1.1: icmp_seq=7 ttl=56 time=52.3 ms
64 bytes from 1.1.1.1: icmp_seq=8 ttl=56 time=105 ms
64 bytes from 1.1.1.1: icmp_seq=9 ttl=56 time=75.5 ms
64 bytes from 1.1.1.1: icmp_seq=10 ttl=56 time=40.3 ms
 
--- 1.1.1.1 ping statistics ---
10 packets transmitted, 9 received, 10% packet loss, time 9033ms
rtt min/avg/max/mdev = 9.503/46.698/104.824/29.420 ms

It works, although 10% packet loss isn’t great. Nonetheless, there is an outbound connection and a return connection. My first guess was DNS, so I tried ping google.com and a few other hosts - all of them failed. Between the successful IP address pings and unsuccessful hostname ones, I think DNS is very likely the culprit at this point.

I opened up my text editor and configured systemd to use a specific DNS server, instead of the ISP assigned ones (I meant to do this earlier anyway, but I forgot):

➤ cat /etc/systemd/resolved.conf
[Resolve]
DNS=9.9.9.9 149.112.112.112 2620:fe::fe 2620:fe::9
FallbackDNS=1.1.1.1
Domains=~.
DNSSEC=allow-downgrade
DNSStubListener=yes

This is basically saying that my DNS should use 9.9.9.9 and 149.112.112.112 as the primary DNS servers (as well as 2620:fe::fe 2620:fe::9 for IPv6. The FallbackDNS entry is used if the primary servers completely fail.

The Domains=~. line is is used to define the routing behaviour for DNS queries. In particular, this line uses the tilde (~) as a negation operator; thus the line says that every query from this machine will go to the globally assigned DNS servers (those set in the DNS line). Along with that, DNSStubListener means that systemd runs a local DNS client (whose address is specified in /etc/resolv.conf). This means that all queries emanating from my computer get sent to this local listener (instead of directly to the DNS servers) before being sent to 9.9.9.9 etc.

Aside

You can use Domains to have split DNS on your computer. For example, if you create another [Resolve] section that write DNS=192.168.1.1 along with Domains=home.lan it means that queries that end with home.lan will use the DNS server in 192.168.1.1, which everything else will go to the global servers.

This worked, since my computer was able to connect to the internet (and the other devices weren’t). Nonetheless, I wanted to know how I’d confirm my new configuration was indeed working. The first step was to use resolvectl status:

➤ resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
    resolv.conf mode: stub
  Current DNS Server: 9.9.9.9
         DNS Servers: 9.9.9.9 149.112.112.112 2620:fe::fe 2620:fe::9
Fallback DNS Servers: 1.1.1.1 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com
                      2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google
          DNS Domain: ~.
Link 4 (wlan0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
     Default Route: yes

To the global configuration is working, but wlan0 (my wifi card) is says the current DNS server is 192.168.1.1 (my local router). This is rather confusing, but, after some digging1, I found it is expected. In short, my wifi card is still receiving and assigning a DNS server via DHCP. This, however, doesn’t matter to me because of the global routing logic - meaning, every query on any domain (because of Domains=~.) gets set to my local DNS client (defined by DNSStubListener=yes). That particular client, then, uses the global configuration and directs its queries to 9.9.9.9, bypassing the local DHCP assigned DNS.

What about the other devices in the network? Well, after all of this worked I confirmed the problem, so I simply logged in to the ISP-provided router and updated the DNS server entries from their own servers to 1.1.1.1 and 1.0.0.12. After a reboot (and many curse words aimed at the crappy interface), all the devices in the networked were able to use the internet.

Footnotes

  1. Yes, this is a pun. dig is a common program to check for these things. ↩

  2. These are Cloudflare servers, instead of the Quad9 ones I used for my own computer. This just because I like to use different ones as a network default and for my main machine. ↩

Graph View