Since moving to IPv6 I have had two recurrent problem: one with some conflict between systemd and the kernel over the /proc/sys/net/ipv6/conf/*/accept_ra, and the second with losing the static ipv6 address assignments on some boxes. I believe the former problem to have something to do with systemd wanting to have control of the sysctl variables, such as accept_ra.

The latter problem is due to the various bits of software that want to have a say in the control of the network. In part some of this is my own fault, as I do have these various bits installed – and if they weren’t installed they could not be causing trouble.

I installed NetworkManager in some places, even when it hadn’t been installed by default, because I wanted to be able to control things with the network applet in gnome. I installed dhclient even though it wasn’t installed, because I wanted better ability to see and control the dhcpv6 leases, particularly the DUID, and network manager made that difficult (and astonishingly, in some cases simply didn’t work).

I eventually came around to the view that I should revise my earlier adopted view of trying to avoid systemd-networkd. At various points I have read some of the angry posts about systemd and in particular about systemd-networkd; things about systemd running roughshod over other parameters. I was annoyed about it, although I didn’t know very much.

I’ve reached a different conclusion now. It seems to me that several things are true:

1. practically everybody, including raspbian, installs systemd now, and relies on it to start up the system;
2. practically everybody installs systemd-networkd, even if it isn’t always enabled;
3. I would like to have only one piece of software trying to control the network, and I know that systemd is going to be involved no matter what – even if I try to avoid systemd-networkd;
4. The NetworkManager applet has very limited capability. But I have a terrible time with dealing with NetworkManager on the command line with nmcli, and I can’t easily just edit text files – I have to use nmcli (I believe I was wrong about this, I think one can edit text files);
5. In some cases I’ve tried using the dhcp clients (dhcpcd and dhclient) to control the dhcpv6 client requirements, particularly the DUID, but it just confuses matters further, as sometimes after a reboot, network manager will just do something different because ultimately each client gins up its own DUID;
6. Systemd-networkd has the virtue that everything is in his text files, and the text files are easy, and every network setting you can imagine is in there; and
7. Finally, the problem I have with systemd wanting to be “in charge” of the sysctl values simply goes away if I use systemd-networkd, as then I will be perfectly happy with it being “in charge”.

So, as an experiment, I am moving to systemd-networkd. The place I most need to do this is oregano, because it is Arch, and (of course, typically with Arch) since one is responsible for picking what bits one wants to use, right now oregano uses a mixed bag of software and loses its dhcpv6 leases at the drop of a hat. I want to switch it to systemd-networkd. However, oregano is my most important box, at least in my home, so I want to experiment first with lavender – a laptop that is also running Arch. If I can get it working on Arch on lavender (which also has the requirement to get the wireless working also), then I can feel confident about doing it on oregano, and later on other boxes.

Note that despite what I used to think, and about which I wrote something stupid in my blog long ago, one shouldn’t conflate systemd-networkd with netplan. Canonical developed netplan, as a front end to either NetworkManager or systemd-networkd. My ubuntu boxes have to cope with the additional complexity of Canonical’s netplan.

Step 1: Get systemd-networkd running on lavender

Lavender is Arch, so I can practice with it before I do it on oregano. Lavender is also wireless, which turned out to present some additional challenges.

First I created a simple .network file for lavender’s wireless interface. The wired interface is the same save for the Name, and having RequiredForOnline=no.

#Lavender - wireless
[Match]
Name=wlp*
[Network]
DHCP=true
MulticastDNS=true
IPv6AcceptRA=true
[Link]
RequiredForOnline=yes
[DHCP]
DUIDType=link-layer-time:2021-07-01 08:31:00
[DHCPv4]
UseDNS=true
[DHCPv6]
RapidCommit=true

It is curious to me that the DUIDType setting is allowed in the DHCP section, and (according to the docs) also allowed in the DHCPv4 section, but is not allowed in the DHCPv6 section. Also note that if you choose a DUIDType of link-layer-time, you must specify a time value, or you get a value of 0. Finally, note that although the docs I saw stipulated section names of DHCPV4 and DHCPV6, the “v” must be lower-case, or a syntax error is flagged.

Setting up wpa_supplicant

N.B. What follows in this section is no longer accurate. There is a later post in which I describe changing over to IWD from wpa_supplicant. I leave this here for completeness, and because I’m lazy, but I have moved 2 or the 3 laptops away from using wpa_supplicant, and will do the other soon. I recommend IWD.

Remember that systemd-networkd doesn’t set up wireless properties. One has to set up wpa_supplicant separately, unless it is already done, and it was not done on Arch. I was unsuccessful with this until I read an article that suggested I tap into the wpa_supplicant@ service. To do that I had to create /etc/wpa_supplicant/wpa_supplicant-wlp13s0.conf and I disabled wpa_supplicant and enabled wpa_supplicant@wlp13s0.

Also, I initially had the priorities backward. A pet peeve – why is priority 1 sometimes the highest, for some things, and near the lowest for others? For wpa_supplicant.conf, higher priority numbers get greater preference so I set the SSID I preferred at 99.

Once wpa_supplicant@wlp13s0 was set up, lavender worked like a champ.

Setting up systemd-networkd on oregano

This worked pretty much out of the box. I used a similar wired file, but without the Required line. I stopped/disabled NetworkManager and started/enabled systemd-networkd, and it allocated a (dynamic) address. Got the client id that it used, put it into the router with a static assignment, and rebooted oregano. It came up with the correct static address. And it did not now lose the static address across a reboot.

I did have to manually remove /etc/resolv.conf and replace it with a symlink to /var/run/systemd/resolve/stub-resolv.conf.

(Later comment): I realized later on that I had failed to do this last step on lavender, and it caused some problems downstream. Most things worked, but oddly there were some cases when dns resolution would fail in odd ways.

Setting up systemd-networkd on lemongrass

Lemongrass, the primary laptop, is running ubuntu. I chose to do it next because I expected some additional challenges with ubuntu, and as with Arch I wanted to do the laptop first before taking on a more “production-like” box.

Initially I used a similar /etc/systemd/network file as I had used for lavender. And I had also set up the wpa_supplicant@wlp110s0 service. This seemed to work, and I initially thought it was ok, but actually I had a mess. Later I found that when I closed the lid there were errors regarding the wpa_supplicant recorded in the log, and after the lid opens again the wpa_supplicant can’t run, trips over an error on a file that should have been removed. This was a giant long-winded troubleshooting effort and I hope I have all the bits and pieces recorded here. I eventually prevailed. But the failure to restart IPv6 after lid event was the initial clue which led into greater issues.

The first and most important fact about ubuntu is that it is using netplan. I was once confused when I first ran into netplan on a server install that it was the same thing as systemd-networkd. Not at all. Netplan is a front end with its own syntax, which it then feeds into either systemd-networkd or NetworkManager. I groped around trying to figure out if there was a way to get rid of netplan, and all I found is that one can go back to ifup/ifdown, but I find no way on ubuntu to use systemd-networkd and NOT use netplan.

As an aside, I am more convinced than ever that I need to migrate my own boxes away from ubuntu to debian. I have done several already. Canonical seems to keep going off on tangents of their own dreaming up stuff to “help me”, which nearly always ends up screwing me. I guess this is another one of those “my own damn fault” things. If you pick a distribution whose primary claim to fame is trying to make things easier for the novice, you shouldn’t complain if they keep trying to implement new “features” which aim to “make things easier”.

Back to the thread, I realized that in my first attempt, I had forgotten that I had to mess with netplan, and so the /etc/netplan file on lemongrass was still calling for NetworkManager. I changed that, and then the dhcpv6 static lease no longer worked – of course, because a different DUID was being used. Initially I was cussing Canonical up and down. I wanted to specify stuff directly to systemd-networkd, and they were forcing me to use netplan; and it turns out netplan has far fewer parameters – I couldn’t specify the DUID in netplan! After some wild goose chases, here is what I am pretty sure is true.

Netplan combines .yaml files from /{lib|run|etc}/netplan/*.yaml, and it arranges the files “lexically” – regardless of which directory they come from. So a file named 01… will be processed before one named 02… regardless of directory. Since /run is rebuilt on boot there normally won’t be files there, but note that files MAY appear in /run/netplan which are created by initramfs, if the initramfs parameters have done some network stuff. Typically I don’t do that and my stuff is all in /etc/netplan, and there is some “background” stuff in /lib/netplan but it isn’t specific to my config.

The .yaml files are parsed in order and depending on the “renderer” specified, files are generated by netplan for systemd-networkd (in /run/systemd/network) or for NetworkManager (in /run/NetworkManager). I will focus on the former case which is where I’m going. So netplan runs, process the .yaml files, and then generates configuration files (.network files) for systemd-networkd. The .network files generated by netplan are in /run/systemd/network. For example, the generated .network files for links and interfaces for the wlp110s0 link, on lemongrass are put into /run/systemd/network/10-netplan-wlp110s0.network.

When systemd-networkd runs, he now has the job of combining – in a similar fashion – .network files instead of .yaml files, but also from /{lib|run|etc}/systemd/network/*.network. These files are for specific links. Also note there is some subtlety to the difference in merging by netplan and the merging by systemd-networkd. In netplan, keys which are processed in later files may amend or override keys in a previously processed file. The “merged” set are used to create the .network file in /run/systemd/network. However, when processing .network files in systemd-networkd, the first link or interface for which a “MATCH” is found will dictate, and further definitions will be ignored even if they match.

So, if I want my hand built “.network” files that I put in /etc/systemd/network to have greater priority/weight/force than the ones generated by netplan, I have to give my files lexically earlier names. If I want to overrule the netplan generated files in /run/system/network/10-netplan-wlp110s0.network, then I have to name my file in /etc/systemd/network something lexically earlier than 10-netplan-wlp110s0.network such as 09-lemongrass-wireless.network.

Ok, I thought, I have it figured out. Wrong. Now there is the issue of the wpa_supplicant, again. It turns out that one can shoot oneself in the foot here also. As discovered with lavender, there may be both a wpa_supplicant.service and/or a wpa_supplicant@<interface>. On ubuntu, to make things even more fun, if you specify the wifi interfaces to netplan (in /etc/netplan/whatever.yaml) then netplan will generate yet another wpa_supplicant. You can have three! What fun. In the end I had to remove from the /etc/netplan/xyz.yaml any mention of the wifi interface to prevent netplan from trying to launch its own wpa_supplicant. And I had to disable the wpa_supplicant, but enable the wpa_supplicant@wlp110s0, and I had to create my definitions in /etc/wpa_supplicant/wpa_supplicant@wlp110s0.conf. I am using the wpa_supplicant which is fired up by networkd, not by netplan, and I am using the one which is specific to a particular interface. Whew.

The lemongrass lid

The second big problem on lemongrass was the IPv6 address would disappear across lid-down, lid-up.

I troubleshot this for hours. I eventually discovered that even though NetworkManager was disabled and stopped, that it was still being launched across the lid-down, lid-up anyway, and it then was running its own wpa_supplicant and the two were fighting. I didn’t attempt to completely troubleshoot what was happening, instead I completely masked NetworkManager, and then it began to work. Once I am pretty confident I have things right, I think I will probably uninstall NetworkManager from lemongrass.

wpa gui

One issue on lemongrass with switching away from NetworkManager is that the gnome applet for NetworkManager goes with it, and there isn’t an applet (that I know of) to control the wpa_supplicant, particularly the adapter specific one.

This means that even though you might have specified several potential SSIDs in your wpa_supplicant-wlp110s0.conf, there is no convenient way to switch between them. It can be done with wpa_cli, but is very inconvenient — there isn’t a simple switch from this one to that one.

Poking around I figured out how to use the wpa_gui package. I installed it on ubuntu. The difficulty with getting it running is permissions. If one wants to run it in the graphical environment, i.e. as a non-root user, once has to arrange for that user to have group privileges on whatever group is able to manage the network device.

When wpa_supplicant runs it establishes a socket for communication, which has to be used by wpa_gui. The specification in /etc/wpa_supplicant/wpa_supplicant@wlp110s0 has to specify where the socket is located in the filesystem, and what the owning group is. So to use wpa_gui, the non-root user one wishes to use to pick SSIDs has to be a member of that group. Stuff on the net talks about making the group wheel, but ubuntu doesn’t have wheel, and uses netdev for the owner of network devices.

Lemongrass is now working correctly as far as I can tell. But I have thought that before. Time will tell.

Cinnamon

Next box to convert was Cinnamon, which is the last of the internal boxes which is obtaining a dynamic lease instead of using the static lease set aside for it. Cinnamon was also one of the boxes upon which the accept_ra value was being lost across reboot.

This is an ubuntu box, so the /etc/netplan needed to be changed to stipulate systemd-networkd. In addition an /etc/systemd/network/ file was created with the same ordinary settings show above, including a DUIDType of link-layer-time.

These changes were made, the new DUID was registered in the router, and Cinnamon came up with the correct address (and with accept_ra properly set, so with a default ipv6 route).