I have written a lot of drivel previously about the various aspects of managing network service, as I’ve tried to figure out the improvements that have been made in linux networking over the last 30 years. I don’t read the kernel threads, and don’t know what’s happening a lot of the time, and haven’t been good about being attentive to release notes, so I get behind and then have to catch up. Previous efforts to catch up have usually been in a hurry when something was busted, and weren’t comprehensive.

This time I have tried to be a bit more comprehensive, so I could settle on the set of solutions and tools I am going to use on the 20 or so different boxes where I have to set this stuff up. A few of them actually need something special, but not most of them. Still, things have to be consistent, and I need to know what I’m doing – at least better than I previously did.

A) Network Configuration and Management systems:

There are basically 3 different systems, that have evolved. Many distros chose one or another for you, but for some (like debian & arch) you have to choose, and can choose any. However, you probably shouldn’t be running both NetworkManager and systemd-networkd at the same time. They are both systemd units you can start, and you can start them both, and it will actually sort of work. But the docs for both say that you should not run then together. I don’t know exactly what evil ensues, and I have in fact actually run both for long stretches of time, but have grown to believe it is probably a bad practice. The three options are:

1. ifupdown system: /etc/network/interfaces, ifup/ifdown scripts. This is the old manual system. Still used in the older pis, before bookworm. You can still choose to use this if you prefer, but it is clunky, error prone, and not feature rich.
2. NetworkManager (ca 2004): Originally Redhat, but adopted by nearly everyone, even the newer Pis (bookworm+). Although possible to use without a gui, NetworkManager is mostly designed for use on user/client systems, desktops and laptops having a graphical interface by way of network control panels, or gnome applets, or such. This is as opposed to use on servers (often headless), or any other non-gui systems. Although it is possible, NetworkManager has a good command line tool, nmcli. NetworkManager is very good as the tool for switching your laptop onto different wifi nets, for example.
3. Systemd-networkd: Newest (ca 2010), and part of systemd. Depends on “.network” files in /etc/systemd/network (and /var and /run) that describe the network in as much detail as you need. Has lots of features for anything you need to do, but is better on servers and headless systems with wired connections, including very complicated connections. Systemd-netword doesn’t have easy and convenient inbuilt ways for switching your laptop around on different SSIDs (for example), or for managing wireless network connection/authentication at all. But it is systemd, well engineered, you can do almost anything with it, it does what it says it will do, and is happy to run roughshod over anybody to get it done.
4. Netplan: I mention this for completeness. This isn’t so much a separate configuration system as a sort of front-end, developed by canonical for ubuntu (so I don’t use it), but it lets you describe the network in yet a third way, and stipulate whether to use systemd-networkd or NetworkManager on the back end. Like a lot of the “we’re here to help you” stuff from canonical it seems like a solution looking for a problem. You get to learn yet another different way to specify the same stuff you would otherwise specify to NetworkManager or systemd-networkd. In fact, appropriately but humorously, for netplan the lingo you use for specifying is in YAML files (which as we all know stands for ‘yet another markup language’), so we have “yet another network configuration language” which is written in “yet another markup language”. At one time I got all snarled up with this because some time ago I had (why I can’t remember) adopted the practice of installing ubuntu server initially, and then adding the desktop later. If you install ubuntu server, canonical makes you set up a netplan. The solution is: don’t do that… if you want ubuntu desktop install it, and I think you just get NetworkManager and no fuss.

B) Wireless Network configuration and authentication

There are two choices in this space for software that can help you figure out what wifi networks are available, and make the association with a wireless interface on your box, including doing the authentication. The two packages are:

1. wpa_supplicant: this is the older and more prevalent system. It uses config in /etc/wpa_supplicant/wpa_supplicant.conf. It can be started with a systemd unit, and there is often a template for starting a copy on a particular interface wpa_supplicant@.service. You can edit the config files by hand, or you can also use the wpa_cli tool.
2. Iwd: newer software, developed by intel. If you install the iwd package you get the iwd daemon, and the iwctl tool for interacting with the daemon.

If you are using one of the network management/configuration tools in A above (systemd-networkd or NetworkManager), you need to tell them if you decide you want it to use iwd instead of wpa_supplicant.

C) Network System Switch (NSS)

The NSS complex is part of the glibc gethostbyname api available in the kernel. Programs/applications mostly use this to get connected to services that turn names of all sorts into addresses, and one subset of that is turning hostnames into addresses (or vice versa). Programs/Services don’t have to use glibc libraries, of course; they can (if they want) do things like try to read /etc/resolv.conf and interact with dns servers/services themselves. But fair to say most dns interactions on linux are probably using the glibc libraries, except maybe some old stuff (people sometimes mention, e.g. old telnet implementations written before the fall of the Roman Empire).

So /etc/nssswitch.conf ends up being very important in deciding what bits of software are getting consulted to figure out dns. I didn’t use to think about this much, barely knew that nsswitch existed. Rather, just used to think that /etc/resolv.conf was the root of everything, but never thought beyond that. No, not so: the NSS config file is crucial.

The config file /etc/nsswitch.conf provides info on finding various “databases” for various kinds of stuff, but for this discussion the set we are interested in is the set called “hosts”, and in the nsswitch.conf file is a single line for hosts, which stipulates the order in which the api should go about consulting various services or bodies of information when confronted with the need to turn a host name into an address. On this line, you stipulate in which order to ask whom. Those you can ask are:

1. files: the /etc/hosts file
2. dns: the dns provider (whoever that is, see discussion below) usually by way of the /etc/resolv.conf file
3. mdns: multicast dns, for use with avahi/zeroconf, ie .local or similar
4. resolve: the systemd-resolved service provided by systemd
5. myhostname and mymachine: both modules from systemd for their implementations of management for vms and the hostname function
6. There are others, more specialized, available which I don’t use.

Two of these above (“files” and “dns”) are provided built-in by glibc, the others are provided by add on “nss modules”, which is a libnss-xx type library that you install. Install the module, if it isn’t installed, and then put its name in the list of modules to which nss successively hands off names for resolution.

An example:

hosts: files mymachines myhostname mdns4_minimal \ [NOTFOUND=return] resolve [!UNAVAIL=return] dns

This example line (a real one), says that for resolving hostnames to ip addresses, glibc libraries should start by looking at the /etc/hosts file (which it knows how to do without an extra module), then consult the mymachines module (a systemd provided one which enables taking better account of containers and vms), then the myhostname module (which again is a systemd provided one that you control with the hostnamectl systemd command, and lets you be more fancy with the system hostname that used to be possible which hostname was just a string in a file). Next the mdns module is consulted, provided by the libnss-mdns module which is part of the avahi package, and this module interacts with the avahi daemon and the avahi system service. Next, module, resolve, is an NSS module for consulting with systemd-resolved which implements very organized and feature-full management of dns with systemd-resolved, if that service is present and turned on. And last, in this example, is “dns”, implemented inbuild within glibc, which seems nominally to mean “consult the dns system”, via the getaddrinfo api (specifically now I think getaddrinfo3). What this means is a function of how you have set up to do dns.

Also, note that there are modifiers like [NOTFOUND=return] that offer some modification of control flow in the presence of some kinds of errors.

D) DNS implementation

If you reach the point where the code going through the nss options reaches the “dns” option, then (my current understanding) is that, for services or programs that use the glibc mechanisms, getaddrinfo will “use /etc/resolv.conf to find a dns server” (my interpretation, quotes mine). In the old days that was as simple as get the address of the server out of /etc/resolv.conf. Now it is more complicated. You can have at least three or four different implementations, you can choose:

1. You can install and implement BIND (or some other dns server) on your box, and you can implement and operate a full scale dns server.
2. You can install something like dnsmasq – a caching stub resolver which doesn’t really have the capacity to actually perform full dns services, but instead communicates with upstream servers who do, and who cache the results (Note well by the way, that glibc/nss/getaddrinfo does not cache any results, fresh lookup every time, which is why a caching stub resolver will help performance).
3. You can install systemd-resolved, which can also be a caching stub resolver, and can also do very complex things including split dns on different interfaces, dnssec, dns over tls, dns over tin cans and barbed wire, whatever you want. Many people hate it, but my take it that like the rest of systemd it is well engineereed, pretty well thought out, paramererized to within an inch of its life, can do anything, but (like Poettering) pretty much no compromise, no accommodation, our way or the highway.
4. You can just let getaddrinfo consult the dns servers in the old fashioned way, as stipulated by the /etc/resolv.conf (but again, no caching). This is what I do.

Depending on what you choose, various stuff may also happen to /etc/resolv.conf, and it may or may not have an actual dns server address in it, and it may or may not be a file, or a symlink. This is a complicated interaction among which dns implementation you are using and what Network Configuration/Management system you are using (NetworkManager or systemd-networkd or neither) and how you instruct that software, if you use it, in its parameters, about how it should manage /etc/resolve.conf. You have parameters and config settings everywhere.

For myself (as of this morning, so could change at any time), my current thinking is that I have very simple requirements here in the house:

• I don’t want to install and manage bind. I may consider doing this on tarragon so I can use the blackhole resolvers, but right now I don’t. The blackhole guys are picky about you using the public resolvers like google or opendns or cloudflare, or amazon in my case, because they can’t measure if you are overusing the services. So right now I have blackhole checking turned off, which results in my getting lower quality spam filtering.
• machines on which I do a lot of web browsing I would like to use the pi-holes, to quash ads, but I’ve had some issues with the piholes recently, so I have stopped the router returning the pi-hole addresses for dns, or sending them out in Ras, and I am manually setting the dns to the pi-hole on juniper, lemongrass and oregano. I don’t do much browsing from anywhere else.
• I would rather not set up dnsmasq (it is set up automatically on cinnamon by the virt-manager or docker or somebody for use on the dummy virtual machine addresses, but not used on cinnamon’s internal interface, and it isn’t used anywhere else). My feeling is that I really don’t have so active an environment that it is hurting me much not to have dns caching, and it is one more piece of software and set of config files I don’t have to manage.
• Every time I have tried to turn on systemd-resolved I end up having some problems with mdns. Systemd-resolved says it has an mdns resolver, which I believe, and a responder, which I believe, and these get consulted (see above). There is also conflicting information about whether resolved and avahi can co-exist. I still have the nss-mdns plugin in and mdns*{_minimal} in the hosts list, and I still have resolve in the hosts list. But (often?/always?) I end up having to troubleshoot if I turn on resolved.
• So systemd-resolved: Do I need it? What for?
  • The only place I worry about split dns is paprika, which has one interface on the vpn router and the other on the internal router. I haven’t done the research to set up resolved to be helpful here, I am simply brute forcing this with the route tables, and use no Ipv6 on that box to avoid leakage.
  • Also, because of the work on Ipv6 I ended up making changes to gai.conf because of the use of ULA addresses in Ipv6, and gai.conf is completely a creature of getaddrinfo, e.g. glibc. If I were to start using resolved, does gai.conf still get consulted about the order to return outbound addresses to use? I think it probably does, but these are the kinds of details that might bite me if I start moving to fancy new systemd-resolved.
  • Other than that, while I believe that systemd-resolved is a well engineered high functionality product, I am too stupid to use it. I’m not using dnssec, or dns over tls, or any of the fancy stuff. I don’t do anything exotic with dns. I’ve used good old getaddrinfo forever, even though I have to confess I knew even less what I was doing than I do now.
  • Ok, I don’t get caching.

D) DHCP implementation

There usually isn’t a reason to care about this too much any more, but for completeness I’ll add this too. A couple of years ago when Mr. G and I did the Ipv6 work, it turned out I had to worry about the DUID, and that required knowing what piece of software was keeping track of DHCP, to ensure that it would use the right DUID, and remember it, and I sometimes had trouble when it (it? Who is “it”? This should probably say when I, stupid operator) would spontaneously change the DHCP software I used, and use a wrong DUID type, and not get the static Ipv6 address, which I cared a lot about. I have grown somewhat sloppy about this – ensuring that I keep the static Ipv6 tables up to date in the router so each of the boxes has the correct address. I’m lazy because it is a pain in the butt to do it manually on Ipv6 on the ubiquiti routers. But if I’m not getting the static address I want it to use, it should be just because I’m sloppy or lazy, and not because I am inadvertantly using the wrong DHCP provider – calling dhclient behind NetworkManagers back or something.

Like other stuff, the selection of DHCP provider software is a function of which network manager you use, and what its configuration options are:

• systemd-networkd can, I believe, use different dhcp clients (like dhcpcd or dhclient) or its own inbuilt dhcp, which it uses by default. I don’t use systemd-networkd much but use the inbuilt dhcp if I do
• ditto with NetworkManager, whose dhcp parameter can be set to dhclient, dhcpcd or internal, and which uses the internal one by default
• It turns out that both of them can quite easily specify everything I need for controlling the DUID, and what sort of DUID to use, so I no long think, as I once did, that I have to use systemd-networkd to control this.
• If using the older pre-bookworm pis, thay use ifupdown and they have dhclient that they use for dhcp (4 and 6).

E) /etc/gai.conf

This is a parameter file for the glibc getaddrinfo3 function, and it turns out to be important because I use Ipv6 ULA Addresses, and I translate those addresses at the router to global unicast addresses. This shenanigan is explained in the Ipv6 post, but is about coping with the possibility that I could experience a prefix change on the global unicast addresses for the boxes, so I want them all using the home controlled ULA addresses internally, and only switching to the global unicast address at the router with snat and dnat, because the router will handle things with the correct prefix even if it changes (which in actuality turns out not to be a problem – it almost never changes after all).

But out of the box gai.conf wants to give lower priority to ULA Ipv6 addresses, so without the mods to the file, getaddrinfo will chose to use IPV4 addresses in preference.

Therefore almost every box has a modified gai.conf, which mostly just says to give the same priority to ULA addresses that it does to GUAs (since I secretly know that the ULAs are going to get turned into GUAs at the router anyway).

F) Managing wireless from command line

I wrote a separate little document about this, because seems like I am often called upon to try to remember how to do this from the command line, particularly on the pis where (on the older ones) there is no NetworkManager, and I don’t usually have a display manager/gui turned on in the gateway pis anyway.

There are two different packages available (maybe there are others). There was the old wireless-tools package, which is now deprecated; and there is the iw package,

Start right off by reminding that earlier we spoke of wireless authentication and the difference between iwd and wpa_supplicant. So there is “iw” and there is “iwd”. Do not confuse the “iw” package, with the “iwd” package. Iwd is an authentication and wireless interface management daemon, written by intel, that replaces wpa_supplicant, and if you install iwd, and its daemon, you talk to it through the iwctl command, just as you use the wpa_cli command to talk to the wpa_supplicant daemon if you use that. Both of them are mostly about the connection and authentication piece of wireless.

The iw package is instead a lower level wireless interface manager. And if you are actually using either iwd or wpa_supplicant, then the iw package, actually cannot (or won’t) help with the connection/authentication piece – it correctly tells you to use the thing you have installed. Mostly the only time I would use iw, now, would be on an older pi.

Mostly, if you find yourself having to configure a new SSID on the command line, you probably should do it with wpa_cli on most boxes, or iwctl on the few that have iwd installed.

G) My chosen setup

So after worrying with this stuff a lot over the years, here is what I currently do:

• I do not use systemd-networkd anywhere except tarragon, which is a headless server in the cloud with no gui interface and no wireless. Everywhere else I use NetworkManaager. I have concluded that anything that I want or need to do in terms of configuration, NetworkManagers connection files can do, in particular, being able to stipulate how to handle Ipv6 DUIDs so I can get static ULA Ipv6 addresses in the house.
• I do not use systemd-resolved anywhere except tarragon. The only reason to do it on tarragon is because it does the caching. And on tarragon I do not have avahi running, so there is not potential for conflict over mdns, and for that matter there isn’t any mdns used on tarragon anyway. I could turn off mdns in systemd-resolved on tarragon, but I haven’t bothered.
• Everywhere else, I use exclusively NetworkManager and I use avahi for mdns, and I have /etc/nssswitch.conf set to make avahi authoritative for the .local/mdns. Although I still have the nss-resolve plugin available, and I still list resolve as one option, in fact since systemd-resolved isn’t running it doesn’t affect the resolution. For dns queries other than .local, I just use the dns inbuild module in nss/glibc, which is simply calling getaddrinfo3 in glibc, and consulting with whatever dns server is specified (which for many boxes is just the router, and for a few where I browse, it specifies the pi-hole).
• For boxes with wireless, I mostly continue to use wpa_supplicant, since that is what is usually out of the box. I have iwd turned on in juniper, paprika and lemongrass I think. I did it on these boxes when I wasn’t as clear about the differences, and I was strugging with wpa_supplicant, and in ignorance decided that iwd would be better (another of the posts which has a lot of drivel in it describes my flawed learings with iwd). IWD may be (probably is), superior, but probably not so much so that I will go to the trouble of installing it. Yet I haven’t gone back to undo it in the cases where I turned it on, and I may not; however I don’t think in my case it provides that much improvement.

So that is my current thinking about which network software to use. This matters mostly on Debian and Arch where I actually have to take steps to ensure I’ve put the right stuff in.