Category Archives: ubuntu

mdadm consistency checks

On ubuntu it seems there is an automatic mdadm array check provided in /etc/cron.d/mdadm, automatically installed with mdadm. This invokes a utility /usr/share/mdadm/checkarray and the cron is set to run this on the first Sunday of every month at 12:57am. And it is set to do this check on all arrays at one time.

This is horrible! So with 5 arrays, totalling 25TB, when this sucker fires up it quickly saturates the i/o capacity of cinnamon, slows to a crawl and settles in to run forever.

I’ve commented that out, and added my own /etc/cron.d/dee_mdadm which doesn’t do all the goofy shenanigans to try to ensure the thing runs on a Sunday (WHY?! Because the guy who wrote it doesn’t work on Sunday?). Instead, my version simply runs on the first of the month, at 12:57am, and on each month it starts the consistency check on a different array. I have 5 arrays, so 3 are checked twice a year, and 2 are checked thrice. Checking just one at a time means there is a good chance it will be done before morning, at least for the small arrays.

I don’t really think the whole consistency check idea is doing me much good, but at least this doesn’t unaccountably bring the system to its knees on the first Sunday of every month.

Update: December, 2021. I haven’t paid much attention to this, and today I found it happening again. I change the arrays around as needed, and hadn’t thought to go recheck what I had done, nor to ensure that the default didn’t reappear on an update, which it did.

For future reference, manually stopping a check:

echo idle > /sys/block/mdx/md/sync_action

or echo check to start manually.

Fail: Resizing /boot on Cinnamon

The current 128GB (119.2GiB) Plextor M.2 disk on cinnamon was created with a very small (256MB/243MiB) /boot partition. Turns out this was a mistake, as it will only hold 3 kernels, so I’m constantly having to go in and remove kernel n-3 to make room.
The remainder of the drive is a single extended partition, containing a 119GiB LUKS container, which contains a 119GiB LVM volume group, which has a root of 107.3GiB and an 11.7GiB swap. The root is only 13% utilized, and while it may grow a little it is in no danger or running out of space. The swap is also far larger than needed. I want to steal a couple of GB from one of them to increase the /boot, but I want to do this while retaining the contents of both the / and the /boot.
Long story short, turns out I wasn’t able to do it. But the journey was a great learning experience for me, so I’ve kept the notes, for future reference.
Continue reading Fail: Resizing /boot on Cinnamon

Disabling ipv6

I was really keen to get ipv6, and very pleased when my provider (wretched hateful Comcast) finally gave me an ipv6 address.

What I didn’t know was that most VPN providers, including the one I use, are only tunneling the ipv4 traffic. So if you make a connection to a location that supports ipv6 (has an AAAA record) the connection will completely bypass the vpn tunnel.

I have temporarily turned off ipv6 on the external interface by putting this into /etc/sysctl.conf:

net.ipv6.conf.enp4s1.disable_ipv6 = 1

Useful kernel parameters

This is just a memory aid.

Most kernel parameters are documented at www.kernel.org

Kernel parameters: doc/html/latest/admin-guide/kernel-parameters.txt

  • boot_delay=n    delay n milliseconds between printk. n<10000
  • console=ttyS1 =ttyUSB0
  • debug                     enable kernel debugging
  • module.dnydbg     see Documentation/dynamic-debug-howto.txt
  • enforcing=0/1  initial SELinux enforcing status
  • ignore_loglevel  ALL printk to the console
  • loglevel=0-7  All printk <n will be printed to console
  • module_blacklist=  comma separated
  • S single mode
  • selinux=0/1 disable selinux
  • apparmor=0/1 disable apparmor

Continue reading Useful kernel parameters

Not starting X Windows

I had some trouble with the graphics drivers, and needed to start the system in runlevel 3. How to do that? Can’t just use the old inittab configuration file and set initdefault to 3 anymore.

I ended up changing the systemd default target. On the running system the /etc/systemd/system/default.target is a symlink to /usr/lib/systemd/system/graphical.target, but if I replace that with a symlink to /usr/lib/systemd/system/multi-user.target, that system will come up to runlevel 3. The easy way to do this is in systemctl

systemctl set-default multi-user-target
systemctl set-default graphical-target

I didn’t do it, but I also found a post recently that said you can simply put a runlevel on the end of the grub command line, i.e. add 3

GTK Tooltips and Eclipse

Eclipse on Fedora (and as I read it, also on Ubuntu) has now got a problem with colors. Specifically, when one hovers over a function to obtain the calling sequence, a pop up appears to provide this information but the background of the pop-up is black, which makes the content unreadable. This is relatively new, though I’m not sure whether it was with the change to Eclipse Neon or earlier, or the change to Fedora 24.

It turns out this is not a setting in eclipse, although I wasted quite a lot of time looking for it. This ends up being a system setting, associated with gtk, the graphical toolkit. If one starts eclipse with

export SWT_GTK3=0;eclipse

which starts eclipse with gtk 2.0, instead of gtk 3.0, the problem goes away, the background of the pop-ups becomes a nice cream color.

The settings we are using are set in the theme one chooses, the default theme being Adwaita. The themes are kept in /usr/share/themes. I tried simply changing to a different theme (TraditionalOk) but the setting affecting this did not change. So I resorted to force.

In /usr/share/themes/TraditionalOk there are directories for gtk-2.0 and gtk-3.0. Within gtk-3.0 I changed the file gtk-widgets.css, specifically, I changed the setting in the tooltip section to background-color #f5f5b5; Previously the value was “@theme_tooltip_bg_color;” and as I read it, that define ought to have the value #f5f5b5. But something is wrong, and the setting is disruptd. But forcing it this way, it seems to work.

Ubuntu Javascript Fail

The “out of the box” apache on Ubuntu comes with a “feature” called “javascript-common” enabled. I haven’t got much idea what this wretched thing does, other than screw me up. I remember now that I had to struggle this some time back on Cinnamon. Now I am in Ohio trying to get something running on a box there, and tearing my hair out once again over the same issue.

This feature adds an Alias directive that takes the directory “/javascript” and sends it off to /usr/share/javascript. So if you are foolish enough to have a directory in your website called “/javascript” (and who would ever dream of putting their javascript files in a directory called javascript, after all) it will fail.

The directory /usr/share/javascript has some nice stuff in it, including jquery, and I guess it is a nice convenience feature for some people. But am I the only one who things it is crazy for a distribution to do something that breaks websites that have a commonly named directory like javascript!?

Correcting Corrupted Kodi Database

I’ve had several cases where one of the Kodi sqlite3 databases becomes corrupted on Coriander. It has happened to Textures, Addons and EPG at different times.  The error that manifests in the Kodi log is The Database Disk Image is Malformed, and once it happens the log is full of this and transactions in the affected database don’t take place.

Did some reading, and discovered that usually you can dump the database, and the dump will be ok.  Try this:

echo '.dump' | sqlite3 fred | sqlite3 new_fred
mv fred bad_fred
mv new_fred fred

Caller Id in Kodi – Redux

I posted in this post about how I arranged to get caller-Id to show up in Kodi. At that time I was still using POTS, but since then I have switched to VoIP and set up Asterisk.

I did find another plugin that would connect to Asterisk and obtain information about incoming calls. But until now it was only half way working. I finally got around to paying attention to it. It was originally provided by “hmronline” and I found it at code.google.com. It was last worked on with Dharma, and I had to do some things to get it ready for Jarvis.

The source code is now in the tarragon svn repository, under the name KodiAsterisk. On coriander, where the production Kodi lives, there is a checked out directory under /Users/wmb called kodiasterisk. For Kodi to install it it has to be zipped, so the sequence is to svn update the directory, then zip the directory as script.service.dee-kodi-asterisk.zip. Then install it within Kodi.

It attaches to asterisk on cinnamon using the Asterisk manager interface.  Authentication on the manager interface uses files in /etc/asterisk/manager.d which has one file per authorized user, containing credentials.

The code has features to

  • give a notification of the caller id
  • put up a picture of the caller
  • pause a video if one is playing
  • redirect the call (within asterisk) if a video is playing
  • indicate whether there are voice mails

I have the notification running, the picture coming up, and the video being paused. I don’t use the redirect feature. Also, since my voice mail is not on Asterisk the features of voice mail aren’t being used.

I did put pictures into a folder within /Storage/Pictures/KodiPictures. The pictures here are named with the phone number, and contain a snapshot of the caller, in jpeg. If there is no picture in the folder for a particular number nothing is put up, but the notification and pausing still occur.

 

Authentication Tokens

I have two websites. The first (on the server tarragon) is readily available on the internet to the public (you are looking at it now), and also has a username/password based login capability. Some selected people are able to get into the back end of this website. Mostly these are people who get their mail on tarragon, or who have websites on tarragon, or both. The login capability allows them to manage their own accounts, change their password, etc.

The second website (on the server oregano) is inaccessible (or at least non-functional) except to authorized users. The authorized users are exactly those people having a login credential on tarragon. The only way to achieve a usable connection to oregano is to first log on to tarragon, and click a link there. This will create a redirect to oregano, passing a token which will allow the connection to succeed. The website on oregano will politely decline to function unless an appropriate token is received.

This article is about building that token. The properties of the token are as follows. First it must provide the identity (username) of the user (the login used on tarragon). It must be encrypted, so that all (or at least part) of its contents are protected. It must not be replayable, that is, it should not be possible for someone to capture the token used by an authorized person, and reuse it later. This includes the provision that it must be time limited, the token should expire after a short time, and subsequently be useless. It should be possible to include other information in the token if needed. For exampe, the same token machinery can be (and is) used for sending an email to handle forgotten passwords, presuming that if joe has forgotten his password, we can send a link to joe’s email address and only joe will receive it.

In the first implementation, I thought to use joe’s password for the encryption. While this can be done, it is really a flawed plan, because I don’t actually have joe’s cleartext password, I only have the hashed version of it. Using joe’s hashed password as a key is obviously vulnerable to capturing the file containing the hashed passwords. The reason they are hashed at all, of course, is that capture of files full of user information occurs all too frequently. If tarragon is available on the internet, I am obliged to assume that a sufficiently motivated and funded attacker could get his hands on the user database. Of course, it is highly unlikely that tarragon would be an interesting target for such an attack. But just because the server doesn’t have national security secrets, that is no excuse to be sloppy.

So I reimplemented it using the certificates on the two machines. Both of the servers have certificates, and use tls for their connections, i.e. they are https instead of http sites. After a little futzing around and reading, I discovered a fairly straightforward way for the php code in server ‘a’ (tarragon) to capture the certificate for remote server ‘b’ (oregano), and extract it’s public key. Then tarragon can encrypt the token using oregano’s public key, so that only oregano can decrypt the token. I could go further, and encrypt again (actually first) using tarragon’s private key, so that oregano could verify that only tarragon could have sent it.

Actually, public key encryption isn’t really used for the token. Instead, a random key is chosen for a symmetric cipher, and the key itself is then encrypted with oregano’s public key, and subsequently decrypted on oregano with his private key. The encrypted key is sent along with the encrypted message.

The function I used in php is part of the openssl library in php, and is called openssl_seal (and the other end is openssl_open. There are lower level functions that would allow one to accomplish the same things, but these seemed straightforward to use. One problem however, is that openssl_seal is written to use RC4 for the cipher. RC4 is frowned upon as insecure in a number of contexts. Openssl_seal allows passing an additional parameter, to select a different cipher, but strangely has no provision for passing an initialization vector so one can’t use any cipher that requires an initialization vector. Eventually I decided to use AES in ECB mode, despite the problems with ECB. This passed syntax but failed horribly at run time – meaning the apache worker just seemed to disappear! In debugging it an error_log call before the openssl_seal was present, and an error_log call afterwards was not, and a surrounding try catch block was not triggered. WTF? It took two days to figure this out. So I just went back to not specifying a cipher and letting it use RC4, and it worked. For the moment at least, I’m leaving it with RC4, since I am only encoding a small token.

The trick to getting a remote certificate in PHP was to use the stream facility, which opens a socket. and a stream context which is a set of parameters to the stream. The stream context is set to use ssl, the socket is established to port 443, and then the stream context will happily yield up the peer certificate that it received during the tls negotiation.