mdadm consistency checks

On ubuntu it seems there is an automatic mdadm array check provided in /etc/cron.d/mdadm, automatically installed with mdadm. This invokes a utility /usr/share/mdadm/checkarray and the cron is set to run this on the first Sunday of every month at 12:57am. And it is set to do this check on all arrays at one time.

This is horrible! So with 5 arrays, totalling 25TB, when this sucker fires up it quickly saturates the i/o capacity of cinnamon, slows to a crawl and settles in to run forever.

I’ve commented that out, and added my own /etc/cron.d/dee_mdadm which doesn’t do all the goofy shenanigans to try to ensure the thing runs on a Sunday (WHY?! Because the guy who wrote it doesn’t work on Sunday?). Instead, my version simply runs on the first of the month, at 12:57am, and on each month it starts the consistency check on a different array. I have 5 arrays, so 3 are checked twice a year, and 2 are checked thrice. Checking just one at a time means there is a good chance it will be done before morning, at least for the small arrays.

I don’t really think the whole consistency check idea is doing me much good, but at least this doesn’t unaccountably bring the system to its knees on the first Sunday of every month.

Updating certificates to “Let’s Encrypt” with ACME

I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.

Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.

But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME

Fail: Resizing /boot on Cinnamon

The current 128GB (119.2GiB) Plextor M.2 disk on cinnamon was created with a very small (256MB/243MiB) /boot partition. Turns out this was a mistake, as it will only hold 3 kernels, so I’m constantly having to go in and remove kernel n-3 to make room.
The remainder of the drive is a single extended partition, containing a 119GiB LUKS container, which contains a 119GiB LVM volume group, which has a root of 107.3GiB and an 11.7GiB swap. The root is only 13% utilized, and while it may grow a little it is in no danger or running out of space. The swap is also far larger than needed. I want to steal a couple of GB from one of them to increase the /boot, but I want to do this while retaining the contents of both the / and the /boot.
Long story short, turns out I wasn’t able to do it. But the journey was a great learning experience for me, so I’ve kept the notes, for future reference.
Continue reading Fail: Resizing /boot on Cinnamon

Mysql 5.7 on Fedora 25

Had a fair amount of trouble getting mysqld to run properly on Fedora 25.

One issue was that mysql 5.7 introduces expiration of passwords, and ALL my passwords expired, including root. I couldn’t do anything, including change my password. Catch 22.

Also, the tables were out of date, and I needed a mysql_upgrade. But I couldn’t do that, because my password was expired.

Eventually figured out that I could change the password with mysqladmin password “goop”

After this, I was able to get in as root, and eventually get all the passwords marked as legitimate.

I was also able to update all the tables with mysql_upgrade.

In researching and troubleshooting this, I discovered a bunch of errors on cinnamon, with innodb generating errors of the form:

InnoDB: Ignoring tablespace `xbmc_video90/directorlinktvshow` because it could not be opened.

I’m pretty sure what happened here was that I removed the files for old xmbc databases from /var/lib/mysql, but didn’t actually drop the databases first. This is a bad habit left over from pre-Innodb when you could get rid of a mysql database just by removing the files.

I made a lot of attempts to fix this with setting in mysqld.conf:

innodb_force_recovery = 1 (or 2 or 3)

but it never worked. In the end I managed to get rid of these errors by :

mysqladmin create xbmc_videoxx (or xbmc_musicxx)

and then

mysqladmin drop xbmc_videoxx

mysqladmin generates a whole bunch of errors and warnings about not being able to find the files, but it does seem to remove the dreck from the innodb files.



Upgrading to Fedora 25

Every upgrade is a little easier, as my restore scripts get better. This was probably the easiest in more than a decade.

This earlier post describes how I put the persistent parts of the system onto btrfs subvolumes, so that on a new system I only need to symlink most of the stuff.

I created a usb stick, containing the resore script, a keyfile for the encrypted drive containing the btrfs subvolumes (the drive is encrypted), and the Nvidia driver installer (which I didn’t need). Halted the system, changed to a different boot ssd drive in the 2.5″ removable drawer, and installed onto it from the downloaded Fedora 25 installer.

This time I made the /boot partition much larger, so that I’m not having to clean up old kernels so often.
Continue reading Upgrading to Fedora 25

Disabling ipv6

I was really keen to get ipv6, and very pleased when my provider (wretched hateful Comcast) finally gave me an ipv6 address.

What I didn’t know was that most VPN providers, including the one I use, are only tunneling the ipv4 traffic. So if you make a connection to a location that supports ipv6 (has an AAAA record) the connection will completely bypass the vpn tunnel.

I have temporarily turned off ipv6 on the external interface by putting this into /etc/sysctl.conf:

net.ipv6.conf.enp4s1.disable_ipv6 = 1


I wanted to know what sort of traffic was being generated by the various websites I host.  So I set up webalizer. Every time I decide to use this, I have to figure it all out again.

Created directory /etc/webalizer.d/ with an entry for each website, e.g. /etc/webalizer.d/fred for, containing a copy of the /etc/webalizer.conf sample file, with the name of the log file changed and the output going into /var/www/usage/fred

Created backup_scripts/webalizer containing:

for W in /etc/webalizer.d/*.conf; do /usr/bin/webalizer -c $W; done

Started the script from crontab.

In apache, I have a 443 virtualhost for which gives me the statistics for website fred when invoked with

Useful kernel parameters

This is just a memory aid.

Most kernel parameters are documented at

Kernel parameters: doc/html/latest/admin-guide/kernel-parameters.txt

  • boot_delay=n    delay n milliseconds between printk. n<10000
  • console=ttyS1 =ttyUSB0
  • debug                     enable kernel debugging
  • module.dnydbg     see Documentation/dynamic-debug-howto.txt
  • enforcing=0/1  initial SELinux enforcing status
  • ignore_loglevel  ALL printk to the console
  • loglevel=0-7  All printk <n will be printed to console
  • module_blacklist=  comma separated
  • S single mode
  • selinux=0/1 disable selinux
  • apparmor=0/1 disable apparmor

Continue reading Useful kernel parameters

Not starting X Windows

I had some trouble with the graphics drivers, and needed to start the system in runlevel 3. How to do that? Can’t just use the old inittab configuration file and set initdefault to 3 anymore.

I ended up changing the systemd default target. On the running system the /etc/systemd/system/ is a symlink to /usr/lib/systemd/system/, but if I replace that with a symlink to /usr/lib/systemd/system/, that system will come up to runlevel 3. The easy way to do this is in systemctl

systemctl set-default multi-user-target
systemctl set-default graphical-target

I didn’t do it, but I also found a post recently that said you can simply put a runlevel on the end of the grub command line, i.e. add 3

GTK Tooltips and Eclipse

Eclipse on Fedora (and as I read it, also on Ubuntu) has now got a problem with colors. Specifically, when one hovers over a function to obtain the calling sequence, a pop up appears to provide this information but the background of the pop-up is black, which makes the content unreadable. This is relatively new, though I’m not sure whether it was with the change to Eclipse Neon or earlier, or the change to Fedora 24.

It turns out this is not a setting in eclipse, although I wasted quite a lot of time looking for it. This ends up being a system setting, associated with gtk, the graphical toolkit. If one starts eclipse with

export SWT_GTK3=0;eclipse

which starts eclipse with gtk 2.0, instead of gtk 3.0, the problem goes away, the background of the pop-ups becomes a nice cream color.

The settings we are using are set in the theme one chooses, the default theme being Adwaita. The themes are kept in /usr/share/themes. I tried simply changing to a different theme (TraditionalOk) but the setting affecting this did not change. So I resorted to force.

In /usr/share/themes/TraditionalOk there are directories for gtk-2.0 and gtk-3.0. Within gtk-3.0 I changed the file gtk-widgets.css, specifically, I changed the setting in the tooltip section to background-color #f5f5b5; Previously the value was “@theme_tooltip_bg_color;” and as I read it, that define ought to have the value #f5f5b5. But something is wrong, and the setting is disruptd. But forcing it this way, it seems to work.