GTK Tooltips and Eclipse

Eclipse on Fedora (and as I read it, also on Ubuntu) has now got a problem with colors. Specifically, when one hovers over a function to obtain the calling sequence, a pop up appears to provide this information but the background of the pop-up is black, which makes the content unreadable. This is relatively new, though I’m not sure whether it was with the change to Eclipse Neon or earlier, or the change to Fedora 24.

It turns out this is not a setting in eclipse, although I wasted quite a lot of time looking for it. This ends up being a system setting, associated with gtk, the graphical toolkit. If one starts eclipse with

export SWT_GTK3=0;eclipse

which starts eclipse with gtk 2.0, instead of gtk 3.0, the problem goes away, the background of the pop-ups becomes a nice cream color.

The settings we are using are set in the theme one chooses, the default theme being Adwaita. The themes are kept in /usr/share/themes. I tried simply changing to a different theme (TraditionalOk) but the setting affecting this did not change. So I resorted to force.

In /usr/share/themes/TraditionalOk there are directories for gtk-2.0 and gtk-3.0. Within gtk-3.0 I changed the file gtk-widgets.css, specifically, I changed the setting in the tooltip section to background-color #f5f5b5; Previously the value was “@theme_tooltip_bg_color;” and as I read it, that define ought to have the value #f5f5b5. But something is wrong, and the setting is disruptd. But forcing it this way, it seems to work.

Ubuntu Javascript Fail

The “out of the box” apache on Ubuntu comes with a “feature” called “javascript-common” enabled. I haven’t got much idea what this wretched thing does, other than screw me up. I remember now that I had to struggle this some time back on Cinnamon. Now I am in Ohio trying to get something running on a box there, and tearing my hair out once again over the same issue.

This feature adds an Alias directive that takes the directory “/javascript” and sends it off to /usr/share/javascript. So if you are foolish enough to have a directory in your website called “/javascript” (and who would ever dream of putting their javascript files in a directory called javascript, after all) it will fail.

The directory /usr/share/javascript has some nice stuff in it, including jquery, and I guess it is a nice convenience feature for some people. But am I the only one who things it is crazy for a distribution to do something that breaks websites that have a commonly named directory like javascript!?

Correcting Corrupted Kodi Database

I’ve had several cases where one of the Kodi sqlite3 databases becomes corrupted on Coriander. It has happened to Textures, Addons and EPG at different times.  The error that manifests in the Kodi log is The Database Disk Image is Malformed, and once it happens the log is full of this and transactions in the affected database don’t take place.

Did some reading, and discovered that usually you can dump the database, and the dump will be ok.  Try this:

echo '.dump' | sqlite3 fred | sqlite3 new_fred
mv fred bad_fred
mv new_fred fred


A next step for providing mail service for some of my users is a primitive webmail facility. For this I used a product called squirrelmail.

Squirrelmail installs, on fedora, into /usr/share/squirrelmail, with config in /etc/squirrelmail. /usr/share/squirrelmail/config contains a perl script,, which provides a simpler configuration experience with explanation of all the somewhat cryptically named fields.

As part of doing this I obtained certificates for all the domains I am hosting – at least all of them over which I have enough control to meet the validation requirements of, meaning either I am postmaster or the documentroot of the website is on tarragon. Without this I would have to choose between not encrypting the mail, or having the user have to put up with mail programs complaining about the certificate.

Caller Id in Kodi – Redux

I posted in this post about how I arranged to get caller-Id to show up in Kodi. At that time I was still using POTS, but since then I have switched to VoIP and set up Asterisk.

I did find another plugin that would connect to Asterisk and obtain information about incoming calls. But until now it was only half way working. I finally got around to paying attention to it. It was originally provided by “hmronline” and I found it at It was last worked on with Dharma, and I had to do some things to get it ready for Jarvis.

The source code is now in the tarragon svn repository, under the name KodiAsterisk. On coriander, where the production Kodi lives, there is a checked out directory under /Users/wmb called kodiasterisk. For Kodi to install it it has to be zipped, so the sequence is to svn update the directory, then zip the directory as Then install it within Kodi.

It attaches to asterisk on cinnamon using the Asterisk manager interface.  Authentication on the manager interface uses files in /etc/asterisk/manager.d which has one file per authorized user, containing credentials.

The code has features to

  • give a notification of the caller id
  • put up a picture of the caller
  • pause a video if one is playing
  • redirect the call (within asterisk) if a video is playing
  • indicate whether there are voice mails

I have the notification running, the picture coming up, and the video being paused. I don’t use the redirect feature. Also, since my voice mail is not on Asterisk the features of voice mail aren’t being used.

I did put pictures into a folder within /Storage/Pictures/KodiPictures. The pictures here are named with the phone number, and contain a snapshot of the caller, in jpeg. If there is no picture in the folder for a particular number nothing is put up, but the notification and pausing still occur.


Saslauth, mail and realms

This server (tarragon) runs a postfix instance which provides mail service for my own as well as for about a dozen other domains belonging to friends and clients. Postfix offers three different ways that a server can receive (be the final destination for) mail directed to a domain:

1) as what postfix calls the canonical destination (i.e. mail for where tarragon IS, and each mail recipient maps onto a user who has a login account on the server, and messages are delivered to that account;

2) as a virtual alias destination, where mail directed to is accepted, but for each such address there is a corresponding forward address to some other location or something, and the actual mail messages do not reside on the server; and finally

3) as a virtual mailbox destination, where mail directed to arrives and is stored in mailboxes on tarragon, awaiting pickup/reading by the user, but without requiring that there be a user z with an actual login account on tarragon. This requires that the mail store on tarragon be set up to maintain different sets of mailboxes for different domains. There can be a user and another user and the mail is not intermixed.

Tarragon uses cyrus-imap as the mail store, and it provides the ability to have different mailboxes for different domains. To support that, the mailboxes are actually constructed differently, so that cyrus-imap can have a mailbox fred, but can also have a mailbox

This requires, in turn, that the imap server be able to identify the correct mailbox when a mail client attaches, and be able to separately authenticate for each mailbox. When cyrus-imap is configured to support this separation, it requires that the username on login be, rather than simply fred.

Cyrus-imap uses the saslauthd daemon to authenticate, and saslauthd in turn calls upon pam, passing in the username,  password and realm (domain) received from imap (or postfix for smtp, or apache for website auth), who receives it in the login message from the user’s mail client. Pam’s authentication for mail is set to use a module called pam_mysql, which is able to match against credentials in a mysql database.

Here is where things get tricky. Take the mail account There is also a user dee with a system account (i.e a type 1 canonical mail account I can choose either to have a) only one entry in the database, for user dee, with a password. That same entry is consulted for access to either mailbox ( or but the are still separate mailboxes. Or alternatively, b) I can have different database entries for dee and, each with its own password. 

A digression: I could, and for many years did, choose to list as a canonical final destination in postfix. If I do that, then mail for goes into the mailbox for user dee on tarragon, just the same as mail for They go into the same mailbox. But when I began supporting virtual mailbox domains, I separated them – using the gray geek account as a test case for hosting virtual mailbox domains. 

Originally I set it up with the idea that there would be seapate database entries. With them separated, an imap login for user dee at host will attach to the mailbox for the user dee, while an imap login for user at host will attach to the mailbox for user who does not have an account on tarragon.

I set all this up over a year ago, and it seemed to be working. Then I set up a new account for a friend who had a new domain name. And I discovered that I had a problem. It so happened that in every case where I had created a mailbox of the form, I actually also had an account fred on, many of those never used and left over from the days when I was only doing canonical logins. I discovered that even though I had entries in the database for login as, the login process was actually using the database entry for fred. It so happened that all these accounts (fred and fred@) had the same password. As soon as I added an account bob@ which did NOT have a corresponding server login account bob, with the same password, it failed.

When I tried using testsaslauthd -u -p <pw> it would work, so the pam machinery and the pam_mysql plugin were working right. The problem occurred between cyrus-imap and saslauthd. I discovered that (a) cyrus imap takes an incoming username of a@b and separates it into “username” and “realm”, and passes those separately to saslauthd, and (b) saslauthd has a parameter ‘-r’ which I had previously failed to discover, which causes it to append the incoming ‘realm’ to the incoming ‘username’ when it attempts to authenticate. Without the ‘-r’ parameter, saslauthd was using only the incoming ‘username’ – fred or bob, in its call on pam. If there was such an account and the password matched, saslauthd would succeed, and the connection would be permitted. Note that cyrus-imapd would now actually connect to the correct mailbox – the issue was that it used the wrong database entry to authenticate against.

Once I turned on the ‘-r’ parameter to saslauthd, imap works for both dee and, but I get a problem in smtp.  On the call from smtp->saslauthd->pam_mysql when saslauthd has the -r parameter simply ‘dee’ as the username will not authenticate. The log record coming from pam_mysql indicates the username comes across as ‘dee’, but fails. If I create a database entry for, and set the username on smtp to, it works.

After a couple of weeks, I began to change my mind. The database is used for several kinds of logins. Going this way – with separate logins for and bob meant: I have to have 2 records in the database. Bob could get confused if he changed his password on one but not the other. For some users I could probably just delete the ‘bob’ entry, and force bob to type in his full email address in order to log in for other things. But I decided this was unappealing.

So I have turned off the ‘r’ parameter, and eliminated all the entries that say from the database. Bob must still use his full email address when connecting to imap but for plain logins in other places, he can use just bob. This also means I no longer have to have a separate database record for canonical mail users to do smtp.


Authentication Tokens

I have two websites. The first (on the server tarragon) is readily available on the internet to the public (you are looking at it now), and also has a username/password based login capability. Some selected people are able to get into the back end of this website. Mostly these are people who get their mail on tarragon, or who have websites on tarragon, or both. The login capability allows them to manage their own accounts, change their password, etc.

The second website (on the server oregano) is inaccessible (or at least non-functional) except to authorized users. The authorized users are exactly those people having a login credential on tarragon. The only way to achieve a usable connection to oregano is to first log on to tarragon, and click a link there. This will create a redirect to oregano, passing a token which will allow the connection to succeed. The website on oregano will politely decline to function unless an appropriate token is received.

This article is about building that token. The properties of the token are as follows. First it must provide the identity (username) of the user (the login used on tarragon). It must be encrypted, so that all (or at least part) of its contents are protected. It must not be replayable, that is, it should not be possible for someone to capture the token used by an authorized person, and reuse it later. This includes the provision that it must be time limited, the token should expire after a short time, and subsequently be useless. It should be possible to include other information in the token if needed. For exampe, the same token machinery can be (and is) used for sending an email to handle forgotten passwords, presuming that if joe has forgotten his password, we can send a link to joe’s email address and only joe will receive it.

In the first implementation, I thought to use joe’s password for the encryption. While this can be done, it is really a flawed plan, because I don’t actually have joe’s cleartext password, I only have the hashed version of it. Using joe’s hashed password as a key is obviously vulnerable to capturing the file containing the hashed passwords. The reason they are hashed at all, of course, is that capture of files full of user information occurs all too frequently. If tarragon is available on the internet, I am obliged to assume that a sufficiently motivated and funded attacker could get his hands on the user database. Of course, it is highly unlikely that tarragon would be an interesting target for such an attack. But just because the server doesn’t have national security secrets, that is no excuse to be sloppy.

So I reimplemented it using the certificates on the two machines. Both of the servers have certificates, and use tls for their connections, i.e. they are https instead of http sites. After a little futzing around and reading, I discovered a fairly straightforward way for the php code in server ‘a’ (tarragon) to capture the certificate for remote server ‘b’ (oregano), and extract it’s public key. Then tarragon can encrypt the token using oregano’s public key, so that only oregano can decrypt the token. I could go further, and encrypt again (actually first) using tarragon’s private key, so that oregano could verify that only tarragon could have sent it.

Actually, public key encryption isn’t really used for the token. Instead, a random key is chosen for a symmetric cipher, and the key itself is then encrypted with oregano’s public key, and subsequently decrypted on oregano with his private key. The encrypted key is sent along with the encrypted message.

The function I used in php is part of the openssl library in php, and is called openssl_seal (and the other end is openssl_open. There are lower level functions that would allow one to accomplish the same things, but these seemed straightforward to use. One problem however, is that openssl_seal is written to use RC4 for the cipher. RC4 is frowned upon as insecure in a number of contexts. Openssl_seal allows passing an additional parameter, to select a different cipher, but strangely has no provision for passing an initialization vector so one can’t use any cipher that requires an initialization vector. Eventually I decided to use AES in ECB mode, despite the problems with ECB. This passed syntax but failed horribly at run time – meaning the apache worker just seemed to disappear! In debugging it an error_log call before the openssl_seal was present, and an error_log call afterwards was not, and a surrounding try catch block was not triggered. WTF? It took two days to figure this out. So I just went back to not specifying a cipher and letting it use RC4, and it worked. For the moment at least, I’m leaving it with RC4, since I am only encoding a small token.

The trick to getting a remote certificate in PHP was to use the stream facility, which opens a socket. and a stream context which is a set of parameters to the stream. The stream context is set to use ssl, the socket is established to port 443, and then the stream context will happily yield up the peer certificate that it received during the tls negotiation.

Setting up OpenVAS

I haven’t done any serious security scanning since playing around with Nessus back in 2006. I decided that I needed to do this, not only on my own servers but on those that I managed for others. It would be very embarrassing to get hacked.

So first I grabbed up Nessus, and discovered that it has, in the meantime, become a mostly commercial product. There is an open source spin called OpenVAS. This is about setting that up.

OpenVAS itself has two parts, and it comes with a third part from a company called Greenbone Security which is a web frontend. The two parts of openvas are the scanner (openvassd) and the manager (openvasmd) while the front end is gsad.

I installed them with dnf, as they are packaged with fedora. This creates a dozen bin files, an /etc/openvas & /etc/pki/openvas, a /var/lib/openvas, and systemd scripts. A good way to go through the setup process is to use openvas-check-setup, which will give clues to what you should do next.

First step was openvas-mkcert which builds a self-signed cert in /etc/pki/openvas. Next step was to install a “redis” server (dnf install redis), and fix its config file with unixsocket /tmp/redis.sock. systemctl enable redis; systemctl start redis.sock. Another step that is needed before downloading the “nvt” files, is to set up a gpg key. Some of the instructions wanted the gnupg directory in /etc/openvas, but the fedora install creates a gnupg directory in /var/lib/openvas, so I used that. Then I downloaded the nvt files (openvas-nvt-sync) and also did openvas-scapdata-sync and openvas-certdata-sync.

I was unsuccessful with checking for signed nvt scripts on either tarragon or oregano. The openvassd scanner (systemctl start openvas-scanner) won’t run with the parameter in /etc/openvas/openvassd.conf set to check the script sigs.

When the instructions said to start the scanner, and then run the manager with the –rebuild option, I started the scanner with systemctl but did the manager with openvasmd –rebuild, to build the “tasks” database.

After that I enabled and started openvas-manager and openvas-gsa (had already enabled and started openvas-scanner).

To use this on tarragon I use an ssh tunnel rather than opening up another port there. It must be connected to using https.

Making MySQL serve UTF8 correctly

If the MySQL server decides that its default environment is UTF8, and that its client actually wants Latin1, it will translate the return values.

I’ve never before had to be careful of the distinction. Perhaps once in a blue moon I would have a record with a “real” quotation mark or a character with an accent, but if it didn’t work correctly, it was never much of a bother.

Once it became important, I had to understand what was happening. I have a table that has filenames in it, and some of those filenames contain characters (a acute, e grave, o umlaut, etc). The actual files on disk have the names encoded in utf8. The records in the database are also recorded in utf8. But the records were being translated by mysql from utf8 to latin1 as they came in. So “Mamá” was recorded on disk as Mam\xc3\xa1 in the directory, and in the database, but when I got the row into memory, the filename field said Mam\xe1. The difference between latin1 and utf8 for this purpose, is that all these many “western/latin special characters” were actually mapped in latin1 to values within the 256 characters available with 1 byte. So the first 128 in the latin1 codespace were ordinary ascii, and the high order 128 had as many of the western/latin diacriticals as possible crammed in there. And in latin1, e9 is a-acute.

But on the web these days, utf8 is much preferred. Latin1 is ok if all you want is the carefully selected subset of 128 characters that can be shoehorned into the high end of the code-point space. But utf8 is a far more general solution. Using a multibyte sequence to represent over a million characters and special symbols.

Turns out mysql has a bunch of variables to control character set and collating sequence. With phpmyadmin, one can look at database->Variables and see characters_set_database, character_set_filesystem, character_set_result, character_set_server, character_set_system, and a bunch more. Or in mysql client one can show varaiables like ‘%character_set%’;

My problem was that the server had come up believing that some of these were set to utf8 and some were set to latin1. I haven’t tried to figure out the logic of how it figures out its default – I don’t want it to default, I want to tell it what I want. So the solution was to add the directive: “character-set-server=utf8” to the mysql configuration file (on cinnamon it was /etc/mysql/mysql.conf.d/mysqld.conf. After restarting all of the relevant character_set_xxx variables come up as utf8.

Saving YouTube Video

Trying to collect the old episodes of NYPD Blue, and there were some I couldn’t find on usenet. Turns out that the DVDs past season 4 were never produced. But there are old episodes, mostly captured from VHS that are out on Youtube.
After a little research, there is a cute way to capture the video files from youtube.
Click the episode on youtube to start it, and then capture the url in the browser location bar.
Now open VLC, select to play a network stream, and give it that url. It will start to play.
At this point, one can just tell VLC to convert and save, but that takes a long time. An easier and faster technique, as long as the youtube webm format is ok:
In vlc, with the stream playing, open tools->codec information and copy the link that is in the “Location” text box at the bottom.
Paste that link into a browser and go, and now the video plays directly from the Youtube servers.
Now you can just right click on the browser window where it is playing, and save video as.