Category Archives: backup

Onedrive

Today I turned on onedrive on my Windows box. Actually two windows boxes, one 10 and one 11. And also, to be accurate, on the Windows 10 box I had to go to some trouble to actually reinstall onedrive which I had long ago completely removed in a fit of pique.

Most of the stuff I write in this blog is about technical stuff I have figured out, and want to record so I can read about what I did later on after I have inevitably forgotten it. This is different. This is more like a cathartic confession, hopefully with the side-effect of purging myself of an unhealthy attitude I have. My confession is that I am unfairly biased against Microsoft, and I tend to automatically reject as unworthy almost anything they create, and I do so sometimes without giving it a fair shake.

Here is the quintessential example. Onedrive is (IMHO) yet another in a long line of ideas which Microsoft rips off from Apple. E.g., in the recent past, Apple created Timemachine, whereupon Microsoft introduces Filehistory. Then Apple creates iCloud, and thereafter Microsoft introduces Onedrive. I don’t love Apple, but that is for industrial policy reasons. When it comes to innovation, though, one has to admit that Apple is constantly innovating. Apple is a visionary innovative engineering company excited by new ideas, which happens also to be excellent at marketing. Microsoft, in my opinion, is a marketing company which is seldom innovative and mostly copies (or buys) other people’s ideas.

I don’t really use iCloud very much, but mostly because I don’t do much actual work on any Macs, and because I don’t take a lot of pictures or listen to a lot of music. I have the default free amount of iCloud storage, but don’t pay for more.

But the important point is that I don’t “resent” iCloud. It doesn’t make me angry. It is there, and like most Apple engineered products it is somewhat seamless. On the other hand, I resented One Drive. I resent that it superimposes itself on Windows, and cannot be ignored. I resent that it is always whining about my not having logged in, and nagging me. I resent most of all that it actually changes the way the file explorer/file system works — once One Drive was implemented, what I used to know about how my friends files were set up became wrong. So typical of Microsoft… they know best, they change things fundamentally and the user must adapt. Reminds me of the old jokes about AT&T pre-breakup. Rule 1: We are AT&T, we don’t care, we don’t have to, we are AT&T. Rule 2: Screw you, see rule 1.

But as time has gone on, I have stopped tilting at this particular windmill. Microsoft may have copied the idea from Apple, but it does make a lot of sense in some ways, like some other changes in the Microsoft experience. I have begun to use a Microsoft account for login on Windows boxes. That also, for a long time I refused to do. And I am letting the Windows boxes do their backup on Onedrive. Because — leaving aside the uber-paranoid reactions that many of us feel about all the big companies having all our data, the fact is that this is a far easier way to do backup. It is a far easier way to reinstall a new machine. All the mumbo-jumbo one used to have to do with windows license keys and licenses for other microsoft products (if one is so unfortunate and foolish to still be paying them for Office for example, which I am not) — all that stuff becomes automatic.

So, grumble, grumble… I am signing my windows boxes in with a microsoft account, and I am letting them back up on onedrive. Grumble, grumble.

Protecting ssh

I have a dozen or so boxes, mostly little raspberry pis, out in people’s houses which let me do backups for them, and attach to their networks. I’ve documented this before in “Gateway pi”, “Memory on the Gateway Pi”, and “Timemachine on Gateway pi” for example.

Connection between these boxes and my house is with SSH, and I use openssh certificates as described in “Using openssh certificates” and “Re-signing Openssh Certificates”. However, there has always been a little nagging problem, which is that these boxes must (re-)establish their connection to me automatically upon reboot, without user intervention. This means that the private keys that accompany the certificates cannot be encrypted, for that would require human intervention.

So there is a risk. Those raspberry pis have upon them a certificate and a private key which would enable access to boxes in my house. Not completely unrestricted access, and not root access, but nevertheless.

Continue reading Protecting ssh

Attaching and backing up the iphone

I have an iPhone 11. From time to time it would be nice to be able to attach it to my network. Always a struggle.

The old Macbook Pro can only run High Sierra, and then only with some special jiggery-pokery. I can sometimes get iTunes on the Macbook to connect to the iPhone, and can usually figure out how to get data into some app using that, or to do a backup, but it is a hassle. The Windows 10 box with iTunes won’t connect to it at all, and (typical of Windows) won’t explain why. I really just want to mount it without all the fuss.

I found a guy on the net who claimed to be able to mount his iPhone on Arch, so I tried following his instructions, which basically involved installing a few libraries usbmuxd, libplist, libimobiledevice and ifuse, the last of which I had to install from AUR. That was easy enough.

Then reboot, plug in the iphone, and voila. It is detected.

I created a directory /ginger, and mounted it with ifuse /ginger, and Bob’s your uncle, I have access to its disk on Arch.

Then I checked on a whim whether I could do a backup. Sure enough libimobiledevice comes with idevicebackup2 which, supposedly, will do a backup of the device. Alas, it doesn’t work, complaining of a protocol mismatch, which according to the net means that the version 1.3.0-3 available on Arch is not the latest, and I need 1.3.1. The option is to download from git and compile from source.

This is low priority for me. I still can do an occasional backup on the Macbook, when I think of it, either locally or to iCloud, via iTunes. The local backup is stored in /Users/dee/Library/Application Support/MobileSync/backup and can be copied elsewhere by root. I don’t actually have much on the iPhone that needs a backup. Many people have their contacts and calendar exclusively on the phone, but I keep both my contacts and calendar in radicale on my server and connect to them from everywhere.

I may eventually do this if there comes a time the backups become important. For now I’ll just wait till a later version shows up in Arch.

Rosemary Recovery 2020

There was a fail event reported on rosemary from one side of a pair of 60GB SSDs, which hold Rosemary_Data. Typical of my installations this mirror set holds the stuff the system needs beyond the os install: /home, databases, certificates, repositories, mail, samba, local bin, etc. Its a mirror set with an encrypted container, containing a btrfs filesystem. The older versions of these setups contain separate btrfs subvolumes for the different directories, newer ones have only one subvolume for that, and another for snaps. This is an old one.

Rosemary doesn’t have an extensive set of services – really only the /home and the databases. No real need for much of anything else. The local bin comes out of the repo anyway, there is no mail, no repository, no certs. However, without that volume the system won’t come up in a usable way. So lesson one learned here was when you get a fail event, attend to it. I let it go for a few days, because I knew I was going to have to pull the case out of the rack mount to get at the SSDs.

Continue reading Rosemary Recovery 2020

Timemachine on Gateway pi

Some people for whom I provide some kinds of support with gateway pis, use Macs. For the pc folk – at least for those on Windows 10, I’ve been seting up to do the filehistory thing, and putting the filehistory onto the /backup drive on the gateway pi. Then it gets sent here overnight. I wanted to do the same for the folks who have Macs, of which there are several.

Continue reading Timemachine on Gateway pi

Out of space on btrfs

I don’t know if I actually know enough to write this post. But I want to record what little I do know about this.

The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most of the dynamic parts of the system. The root volume ‘/’ is very small, about 20GB. Just enough to install the Centos code and keep a few little things. The great majority of the information needed to run the system is symlinked out of /, which includes /home, mail, databases, websites and their data, the repositories, certificates, local scripts etc.

This is a 180GB disk, and it currently is running about 55% full, i.e. almost 100GB used. Among the information on this disk are snapshots of all the tarragondata, every night for 30 days. This isn’t disaster backup/disk failure backup (which is elsewhere), this is “operator error” backup.

A couple of weeks ago I began to experience a new kind of failure. In the middle of the night, suddenly this btrfs volume would report that it was out of space – usage 100%, although the amount of storage in use was, still the roughtly 100GB that it normally uses. It manifestly was not actually out of space.

Continue reading Out of space on btrfs

Clamd signatures and Apache memory

After implementing the new tarragon the biggest problem I had involved the clamav package, and its loading of signatures. If clamd doesn’t come up and open its socket, then amavisd (the daemon who is consulted by postfix to handle all the checking of each piece of mail on input and output) will fail (assuming he is configured to do virus checking), This results in various problems. Amavis will mark the mail as “unchecked”, but worse, it will report failure back to postfix who gets confused and very often the message is delivered two or three times.

Clamd, the clamav daemon, now has over 6 million signatures. There are a lot of bad boys out there. The signatures are loaded by clamd from its database (in /var/lib/clamav) on startup, into memory. As a result, clamd has a large memory footprint, almost 800Mb on my system. The first issue, discovered before going live, was that systemd’s default parameters expect any daemon he starts to load within 90 seconds. If it fails to check in within that time, systemd considers it broken and terminates it. Clamd takes at least 3 minutes to load. I had to set a special TimeoutStartSec value in the systemd service script for clamd@.service.

Whew! I thought, boy I’m glad I figured that out. Hah!

Continue reading Clamd signatures and Apache memory

Tarragon Rebuild 2019

This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.

The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.

Continue reading Tarragon Rebuild 2019

Memory on the Gateway Pi

I now have 8 of these gateway boxes out there. This morning as I was checking backups on one of them, I observed that it took quite a long time to respond. I ran a top on it and was horrified to see that its memory use was 100% and so was its swap. Holy @#$%!@ Batman!

Most of the memory was being used by the lxpanel. And (hangs head in embarrassment) there were actually two lxpanels running – one for the console and one in the vnc window I launch at startup.

It seems the lxpanels leak. I don’t know how badly, but it doesn’t matter. These boxes are meant to run forever so even a tiny leak is eventually fatal.

Well this was simple. I will seldom, if ever, need to get into a graphical environment remotely, and if I do I can always start vnc from the command line. So I took out the startvnc from the startup script. And I have even LESS need for a graphical console since there is not even a monitor on these things. So I set the default systemd target to multi-user.target.

Did this on all the gateways that are running on pi-zeros. Those few running on bigger ubuntu boxes I didn’t really have the problem anyway.

After rebooting them they come up with no lxpanels. I’ll watch the memory use, but I think this will fix the problem.

Invisible zfs snapshot directory

I found out today something I am sure to forget.

In every zfs dataset there is an invisible directory (by invisible, I mean that it does NOT show up with ls -a) name .zfs. In side this directory are two subdirectories, shares and snapshots.

The snapshots subdirectory is a perfectly serviceable read-only access to all the snapshots. Viz:

Continue reading Invisible zfs snapshot directory