I now have 8 of these gateway boxes out there. This morning as I was checking backups on one of them, I observed that it took quite a long time to respond. I ran a top on it and was horrified to see that its memory use was 100% and so was its swap. Holy @#$%!@ Batman!
Most of the memory was being used by the lxpanel. And (hangs head in embarrassment) there were actually two lxpanels running – one for the console and one in the vnc window I launch at startup.
It seems the lxpanels leak. I don’t know how badly, but it doesn’t matter. These boxes are meant to run forever so even a tiny leak is eventually fatal.
Well this was simple. I will seldom, if ever, need to get into a graphical environment remotely, and if I do I can always start vnc from the command line. So I took out the startvnc from the startup script. And I have even LESS need for a graphical console since there is not even a monitor on these things. So I set the default systemd target to multi-user.target.
Did this on all the gateways that are running on pi-zeros. Those few running on bigger ubuntu boxes I didn’t really have the problem anyway.
After rebooting them they come up with no lxpanels. I’ll watch the memory use, but I think this will fix the problem.
Cinnamon and Rosemary are now both happily rack-mounted in the basement (where it is cool, and where their many disk drives and fans can make as much racket as they wish).
Mostly I control them from the office with ssh and or vnc, but once in a while I need to actually be down there. My neighbor gave me a monitor, I have plenty of mice and keyboards, so I hooked up a KVM switch on the two of them so I didn’t have to keep getting behind the rack to move the monitor.
But alas, neither of them picked up the resolution of the monitor, I suppose (not sure) that with the KVM in the middle, they can’t really read the EDID and such stuff from the monitor. And since it is an “unknown” monitor, the display panel only shows 1024×768, 800×600 etc. The monitor itself helpfully tells me that it wants to be 1440×900 @60Hz.
Continue reading Forcing Monitor resolution
I found out today something I am sure to forget.
In every zfs dataset there is an invisible directory (by invisible, I mean that it does NOT show up with ls -a) name .zfs. In side this directory are two subdirectories, shares and snapshots.
The snapshots subdirectory is a perfectly serviceable read-only access to all the snapshots. Viz:
Continue reading Invisible zfs snapshot directory
I realized as I was writing a new post that I had never documented the gateway pi undertaking.
This started when a friend in the mountains got a new internet service where the ISP would not allow him (and therefore me) access to his router. As a result I could no longer use ssh to connect to his systems.
I solved this problem by setting his systems up to use a tool called autossh, with which I could have his system start, monitor, and keep running an ssh daemon with reverse tunnels open to my system. I could then reach him by attaching through the reverse tunnels.
Continue reading Gateway pi
I had in mind (still do) to use Cinnamon as a host for virtual machine. In fact, I have had that idea in the back of my mind for many years. Recently that idea percolated up to the top again, and one thing I did was to buy some additional ram for it, I bought a 16G stick and tried to add it. It wouldn’t boot. The very poorly written manual on the motherboard seems to suggest that it absolutely requires one to have balanced sticks in the dimm slots. I find that hard to believe, but decided it couldn’t hurt to comply, and bought another stick, so I would have 2 8G sticks, and 2 16G sticks, 48G.
It still wouldn’t boot. But I noticed that this doesn’t look like the hardware is failing – it gets up into Xen and then stops. I don’t think this is a hardware problem with the memory.
After some googling around I found an article on the Suse website with a similar thing, saying that Dom0 won’t come up if it has more than 32G of memory, and offering a solution.
I’m very ignorant about Xen. I have never really gotten beyond installing it, with my Cinnamon ubuntu installation in Dom0 and using all the resources. But, but it is clear of course that the right way to do this is for the Dom 0 to be small and confined to its management job, and Cinnamon should actually be a Dom U.
What seems to be true is that if you do not specify on the command line, the Dom 0 will come up with all the memory. And that if you have more than 32GB of memory for it to come up with it will fail. Thus if you have more than 32GB of memory, you MUST avail yourself of the command line to limit the memory available to Dom 0.
I added to the linux command line in the default grub,:
And the box came up fine. Once I manage to get Cinnamon and it’s functions into separate Dom U, I will reduce the dom0 down to 1 G or so.
I suddenly had a problem on the laptop (ubuntu 18.04, Bionic), and I’ve seen similar problems on Cinnamon. One manifestation is that when I try to start terminal using the gnome-terminal.desktop app, i.e. the icon in the dock, the launch fails, with no stated reason. A look in the log reveals a problem in Python loading _gi.
I resolved to look at it later. I was able to start the terminal from the desktop by right clicking, and open terminal.
Later I had a similar problem trying to launch gnome-tweak. It failed in the same way, trying to start up the python app it attempted to load _gi.
I have only the vaguest clue what is going on here. I looked at the python code. Indeed they are attempting:
From . import _gi.
And the directory /usr/lib/python3/dist-packages/gi does not have a library named _gi. It does however have two library files:
Perusing the net, I find some suggestion that these libraries, with the 36m designation are somehow specific to or found by Python 3.6, and that if I want this to work under Python 3.7, I have to copy these files, and change 36m to 37m.
Ok, what have I got to lose? I did that simple thing – with just a simple copy in the above named directory. Lo and behold, it fixed the symptom that I was experiencing.
I don’t really like fixing things with magic when I don’t know what is happening, but now is not the time to go on a voyage of discovery with Python.
A few months ago Fedora crashed, and wouldn’t boot. It seems to do that from time to time. I have about had it with Fedora. I have at least three times as much trouble with Fedora as I do with Ubuntu.
So I reinstalled Fedora. I was back-level anyway, as I have grown very cautious of their automatic update – which about a third of the time ends up requiring a full system rebuild. My thinking about it wasn’t quite this black and white, but might has well have been: “It’s going to crash eventually, and require me to scrape it down to the bare metal – I might as well wait till that happens, rather than hastening the process by trying to update fedora.”
Anyway, on that occasion back in May, I rebuilt a new Fedora 30 system on a new disk, and restored everything.
Continue reading Root Account is locked
The topology for the handling of downloads of stuff via a vpn previously involved a vpn client directly on rosemary. The problem with this was that sometimes the vpn would fail – it would get disconnected from the remote end. If I didn’t realize this, and started a download, it would be in the clear.
I thought a better solution was to have a separate router (herein the “vpn” router) between rosemary and the external router, and to have that router establish a constant vpn through it’s wan interface, through the external router. Everything that connected to a lan port on the vpn router would be protected. Rosemary would then use the vpn router as its path to the internet. Everything that rosemary sends or receives from the internet would come exclusively through the vpn router.
Continue reading Adding a VPN Router to the network
The objective of this project was to install a vpn server on one of the boxes in the cloud (initially asafoetida, then moved to tarragon), in order to provide a VPN server service for a friend who was traveling. My friend uses the name Darrell for his client, so in what follows the vpn is called by this name.
Create a Certificate Authority
A lot of the instructions, even from openvpn site, say to use the “easyrsa” package to generate the certificates for openvpn. This package seems to be put out by the openvpn boys, or at least with their cooperation. But I didn’t do that. I created a ca with raw openssl.
Continue reading Setting Up Openvpn Server
II have a terrible time keeping this straight. This is a memory aid. I think it is correct, but it has been assembled from internet sources and not from reading the code, so this is not authoritative.
Continue reading Netfilter table order